bert_text_classification.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.6"
    },
    "colab": {
      "name": "bert_text_classification.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/Deep_Learning_in_LangTech_course/blob/master/bert_text_classification.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2KiiaDvGELay"
      },
      "source": [
        "# Text classification with BERT\n",
        "\n",
        "This notebook briefly demonstrates fine-tuning a pretrained BERT model to a text classification task.\n",
        "\n",
        "You probably want to run this notebook with GPU acceleration, as fine-tuning BERT on CPU can be fairly slow even with a comparatively small dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VbLYhNpSELa7"
      },
      "source": [
        "## Configuration\n",
        "\n",
        "The following variables configure a few aspects of the data, model, and training process. To adapt this example to a different dataset, you'll probably want to change these to match.\n",
        "\n",
        "Note in particular that we're limiting the number of examples and the maximum sequence length to make training faster. \n",
        "\n",
        "When running on GPU, it's also necessary to make sure that the input length and batch size are not so large as to cause a batch to exceed GPU memory. If you're getting a message like `Resource exhausted: OOM when allocating tensor`, try smaller numbers for these."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PJRXYM58ELa8"
      },
      "source": [
        "# Maximum number of examples to read\n",
        "MAX_EXAMPLES = 1000\n",
        "\n",
        "# Maximum length of input sequence in tokens\n",
        "INPUT_LENGTH = 250\n",
        "\n",
        "# Number of epochs to train for\n",
        "EPOCHS = 3\n",
        "\n",
        "# Optimizer learning rate\n",
        "LEARNING_RATE = 0.00002\n",
        "\n",
        "# Training batch size\n",
        "BATCH_SIZE = 8"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qlOi4EtdELa8"
      },
      "source": [
        "## Package setup\n",
        "\n",
        "We'll use [keras-bert](https://github.com/CyberZHG/keras-bert). Make sure the package is installed. (`pip` is the Python [package installer](https://pip.pypa.io/en/stable/))"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eg8Y85ztELa9",
        "outputId": "5c99bf08-2040-4891-c8bc-08b04d6f6552"
      },
      "source": [
        "!pip3 install keras-bert"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: keras-bert in /Users/smp/Library/Python/3.7/lib/python/site-packages (0.81.0)\n",
            "Requirement already satisfied: numpy in /Users/smp/Library/Python/3.7/lib/python/site-packages (from keras-bert) (1.18.1)\n",
            "Requirement already satisfied: Keras in /Users/smp/Library/Python/3.7/lib/python/site-packages (from keras-bert) (2.3.1)\n",
            "Requirement already satisfied: keras-transformer>=0.30.0 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from keras-bert) (0.32.0)\n",
            "Requirement already satisfied: scipy>=0.14 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from Keras->keras-bert) (1.4.1)\n",
            "Requirement already satisfied: six>=1.9.0 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from Keras->keras-bert) (1.14.0)\n",
            "Requirement already satisfied: h5py in /Users/smp/Library/Python/3.7/lib/python/site-packages (from Keras->keras-bert) (2.10.0)\n",
            "Requirement already satisfied: keras-preprocessing>=1.0.5 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from Keras->keras-bert) (1.1.0)\n",
            "Requirement already satisfied: pyyaml in /Users/smp/Library/Python/3.7/lib/python/site-packages (from Keras->keras-bert) (5.3)\n",
            "Requirement already satisfied: keras-applications>=1.0.6 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from Keras->keras-bert) (1.0.8)\n",
            "Requirement already satisfied: keras-multi-head>=0.22.0 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from keras-transformer>=0.30.0->keras-bert) (0.22.0)\n",
            "Requirement already satisfied: keras-layer-normalization>=0.12.0 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from keras-transformer>=0.30.0->keras-bert) (0.14.0)\n",
            "Requirement already satisfied: keras-position-wise-feed-forward>=0.5.0 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from keras-transformer>=0.30.0->keras-bert) (0.6.0)\n",
            "Requirement already satisfied: keras-embed-sim>=0.7.0 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from keras-transformer>=0.30.0->keras-bert) (0.7.0)\n",
            "Requirement already satisfied: keras-pos-embd>=0.10.0 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from keras-transformer>=0.30.0->keras-bert) (0.11.0)\n",
            "Requirement already satisfied: keras-self-attention==0.41.0 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from keras-multi-head>=0.22.0->keras-transformer>=0.30.0->keras-bert) (0.41.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XquhckmWELa_"
      },
      "source": [
        "## Tensorflow configuration\n",
        "\n",
        "We'll need set an environment variable for keras-bert to use `tensorflow.python.keras`. (This is a technical detail that is not related to the model.)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J4XwkVZwELa_"
      },
      "source": [
        "import os\n",
        "\n",
        "\n",
        "os.environ['TF_KERAS'] = '1'    # Required to use tensorflow.python.keras with keras-bert"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zJtvfHCjELbB"
      },
      "source": [
        "## Download dataset\n",
        "\n",
        "We'll use the IMDB dataset in JSON format. (This should be familiar to you from previous notebooks in this course.)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VQO35xRqELbD",
        "outputId": "87f6af56-a3be-4d62-8b76-ba6def8cd0f4"
      },
      "source": [
        "# Give -nc (--no-clobber) argument so that the file isn't downloaded multiple times \n",
        "!wget -nc https://raw.githubusercontent.com/TurkuNLP/Deep_Learning_in_LangTech_course/master/data/imdb_train.json"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "File ‘imdb_train.json’ already there; not retrieving.\r\n",
            "\r\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-7Bwsi5pELbE"
      },
      "source": [
        "## Load data\n",
        "\n",
        "We load the IMDB data and shuffle it as usual."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BkRq2pW5ELbF",
        "outputId": "6f00297f-d927-442c-8c05-26eccbe40ea1"
      },
      "source": [
        "import json\n",
        "import random\n",
        "\n",
        "\n",
        "with open('imdb_train.json') as f:\n",
        "    data = json.load(f)\n",
        "\n",
        "random.seed(1234)    # This makes the shuffle produce the same order every time\n",
        "random.shuffle(data)\n",
        "\n",
        "if len(data) > MAX_EXAMPLES:\n",
        "    print('Note: truncating examples from {} to {}'.format(len(data), MAX_EXAMPLES))\n",
        "    data = data[:MAX_EXAMPLES]\n",
        "\n",
        "# Look at the data\n",
        "print(type(data))\n",
        "print(type(data[0]))\n",
        "print(data[0].keys())\n",
        "print(data[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Note: truncating examples from 25000 to 1000\n",
            "<class 'list'>\n",
            "<class 'dict'>\n",
            "dict_keys(['class', 'text'])\n",
            "{'class': 'pos', 'text': \"I saw Insomniac's Nightmare not to long ago for the first time and I have to say, I really found it to be quite good. If you are a fan of Dominic Monaghan you will love it. The hole movie takes place inside his mind -or does it? The acting from everyone else is a little rushed and shaky and some of the scenes could be cut down but it works out in the end. The extras on the DVD are just as great as the film, if not greater for those Dom fans. It has tons of candid moments from the set, outtakes and a great interview with the director. Anyone who has gone through making an independent film will love to watch Tess (the director), Dom and everyone else on the very small close personal set try to bang out this little trippy creepy film. It was pretty enjoyable and I'm glad to have it in my collection.\"}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z63VA-bMELbF"
      },
      "source": [
        "Separate the texts and labels into lists of their own."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eJaI7NhvELbG",
        "outputId": "ef7a2c8b-6fe8-453a-b918-8673c043db6a"
      },
      "source": [
        "texts = [example['text'] for example in data]\n",
        "labels = [example['class'] for example in data]\n",
        "\n",
        "# Example text and label\n",
        "print('Text:', texts[0])\n",
        "print('Label:', labels[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Text: I saw Insomniac's Nightmare not to long ago for the first time and I have to say, I really found it to be quite good. If you are a fan of Dominic Monaghan you will love it. The hole movie takes place inside his mind -or does it? The acting from everyone else is a little rushed and shaky and some of the scenes could be cut down but it works out in the end. The extras on the DVD are just as great as the film, if not greater for those Dom fans. It has tons of candid moments from the set, outtakes and a great interview with the director. Anyone who has gone through making an independent film will love to watch Tess (the director), Dom and everyone else on the very small close personal set try to bang out this little trippy creepy film. It was pretty enjoyable and I'm glad to have it in my collection.\n",
            "Label: pos\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1iloppNIELbG"
      },
      "source": [
        "## Download pretrained BERT model\n",
        "\n",
        "As training BERT from scratch generally takes days, we'll here load a pretrained model and fine-tune it for our task. URLs to download pre-trained models made available by Google are found at https://github.com/google-research/bert .\n",
        "\n",
        "`cased_L-12_H-768_A-12` is a case-sensitive BERT \"base\" model for English: 12 layers, 768-dimensional hidden state, and 12 \"heads\" for multi-head attention."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "krlaJmsXELbH",
        "outputId": "7663a039-9dfc-401d-e4ec-1cc6c0c7204a"
      },
      "source": [
        "# Give -nc (--no-clobber) argument so that the file isn't downloaded multiple times \n",
        "!wget -nc https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "File ‘cased_L-12_H-768_A-12.zip’ already there; not retrieving.\r\n",
            "\r\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2HHiPgFpELbH"
      },
      "source": [
        "Unpack the downloaded file:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z4glKjmQELbI",
        "outputId": "4f894566-3135-4e7e-b696-7a401e7c32d2"
      },
      "source": [
        "# Give -n argument so that existing files aren't overwritten \n",
        "!unzip -n cased_L-12_H-768_A-12.zip"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Archive:  cased_L-12_H-768_A-12.zip\r\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tbUU9-gwELbI"
      },
      "source": [
        "There are three key parts to the package contents:\n",
        "\n",
        "* `vocab.txt`: plain text file listing vocabulary items\n",
        "* `bert_config.json`: model configuration in JSON format\n",
        "* `bert_model.ckpt.*`: model checkpoint data with pre-trained weights in [Tensorflow checkpoint format](https://www.tensorflow.org/guide/checkpoint)\n",
        "\n",
        "Take note of the path to these (you'll need to change this if you pick a different model)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zvgesQRKELbJ"
      },
      "source": [
        "bert_vocab_path = 'cased_L-12_H-768_A-12/vocab.txt'\n",
        "bert_config_path = 'cased_L-12_H-768_A-12/bert_config.json'\n",
        "bert_checkpoint_path = 'cased_L-12_H-768_A-12/bert_model.ckpt'    # suffixes not required"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "--tneJcvELbJ"
      },
      "source": [
        "Also take note if the model we downloaded was a case-sensitive (cased) or not. (This must match the model.)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n4l-iZ24ELbJ"
      },
      "source": [
        "model_is_cased = True"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KqJgNQ65ELbJ"
      },
      "source": [
        "## Load BERT vocabulary\n",
        "\n",
        "This is just a plain text file with one vocabulary item per line."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jQgK_nLKELbK",
        "outputId": "db38cd2d-706d-4923-f781-1fb22eee381c"
      },
      "source": [
        "vocab = []\n",
        "with open(bert_vocab_path) as f:\n",
        "    for i, line in enumerate(f):\n",
        "        vocab.append(line.rstrip('\\n'))    # rstrip to remove newline characters\n",
        "\n",
        "\n",
        "# Print a list with every 500th vocabulary item\n",
        "print(vocab[0::500])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['[PAD]', 'щ', '吉', 'told', 'space', 'operations', 'proposed', 'Oxford', 'showing', 'domestic', 'mountains', 'commission', 'voices', 'associate', 'hills', 'Guide', 'relaxed', 'Page', 'Heights', 'singers', 'Interior', 'considers', 'facilitate', 'shouting', '1826', 'constitute', 'alter', 'clip', 'Into', 'Memory', 'ballad', 'Owens', 'Langdon', 'aquatic', 'stereo', 'Cass', 'Shock', '195', '##tec', '##sonic', 'attested', '##rdes', '1840s', '##90', 'Guys', '##rien', 'Munro', 'Ursula', 'mesh', 'diplomacy', 'Newmarket', '##oughs', 'synthesizers', 'Drugs', 'monstrous', '##ynamic', 'troll', '##ٹ']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rsIbSQPYELbK"
      },
      "source": [
        "## Load BERT configuration\n",
        "\n",
        "The configuration is just a JSON file, so we can read it in with `json.load` from the python `json` library.\n",
        "\n",
        "We won't actually need to use these configuration details directly (keras-bert takes care of them for us), so this is just here to show what information is contained in the config file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V5UHGVOKELbL",
        "outputId": "319fbc29-6952-4d11-8e55-4fe49400df19"
      },
      "source": [
        "from pprint import pprint    # pretty-printer for output\n",
        "\n",
        "\n",
        "with open(bert_config_path) as f:\n",
        "    config = json.load(f)\n",
        "\n",
        "\n",
        "# Print configuration contents\n",
        "pprint(config)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'attention_probs_dropout_prob': 0.1,\n",
            " 'hidden_act': 'gelu',\n",
            " 'hidden_dropout_prob': 0.1,\n",
            " 'hidden_size': 768,\n",
            " 'initializer_range': 0.02,\n",
            " 'intermediate_size': 3072,\n",
            " 'max_position_embeddings': 512,\n",
            " 'num_attention_heads': 12,\n",
            " 'num_hidden_layers': 12,\n",
            " 'type_vocab_size': 2,\n",
            " 'vocab_size': 28996}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Al3620LpELbL"
      },
      "source": [
        "## Create BERT tokenizer\n",
        "\n",
        "To create the tokenizer, we'll need a mapping from vocabulary items to their corresponding integer indices. We do this conventionally using `enumerate`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KUlLgeI5ELbL",
        "outputId": "f3ccdb5d-8f69-4702-b092-edfc279e18b8"
      },
      "source": [
        "# Create mapping from vocabulary items to their indices in the vocabulary\n",
        "token_dict = { v: i for i, v in enumerate(vocab) }\n",
        "\n",
        "\n",
        "# Print some random examples of the mapping\n",
        "pprint(dict(random.choices(list(token_dict.items()), k=10)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'##FF': 17515,\n",
            " '##bi': 5567,\n",
            " '##good': 23684,\n",
            " 'Colours': 23189,\n",
            " 'Lockheed': 18904,\n",
            " 'Merrill': 17247,\n",
            " 'Wheel': 19754,\n",
            " '[unused82]': 82,\n",
            " 'clock': 4705,\n",
            " 'corridors': 19507}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O7q3jwGvELbM"
      },
      "source": [
        "We'll use the keras-bert `Tokenizer` for BERT tokenization. The implementation supports\n",
        "\n",
        "* (Optional) lowercasing: `Hello` → `hello`\n",
        "* Basic tokenization: `Hello!` → `Hello` `!`, `multi-part` → `multi` `-` `part`\n",
        "* Wordpiece tokenization: `comprehensively` → `comprehensive` `##ly`\n",
        "* Adding special tokens: `Sentence`  → `[CLS]` `Sentence` `[SEP]`\n",
        "* Mapping to integer indices\n",
        "* Generating segment sequence\n",
        "* (Optional) padding and truncation to length\n",
        "\n",
        "In the following example, notice how words not in the dictionary are broken up into subwords (with continuation parts starting with `##`) and how unknown _characters_ are mapped to a special unknown word token `[UNK]`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dXBiE3zVELbM",
        "outputId": "65a78b3c-0f07-4d00-9644-605b2b357fe4"
      },
      "source": [
        "from keras_bert import Tokenizer\n",
        "\n",
        "\n",
        "tokenizer = Tokenizer(token_dict, cased=model_is_cased)\n",
        "\n",
        "\n",
        "# Let's test that out\n",
        "for s in ['Hello BERT!', 'Unknown: 你']:\n",
        "    print('Original string:', s)\n",
        "    print('Tokenized:', tokenizer.tokenize(s))\n",
        "    indices, segments = tokenizer.encode(s, max_len=20)    # max_len for padding and truncation\n",
        "    print('Encoded:', indices)\n",
        "    print('Segments:', segments)\n",
        "    print('Decoded:', ' '.join(tokenizer.decode(indices)))\n",
        "    print()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Original string: Hello BERT!\n",
            "Tokenized: ['[CLS]', 'Hello', 'B', '##ER', '##T', '!', '[SEP]']\n",
            "Encoded: [101, 8667, 139, 9637, 1942, 106, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            "Segments: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            "Decoded: Hello B ##ER ##T !\n",
            "\n",
            "Original string: Unknown: 你\n",
            "Tokenized: ['[CLS]', 'Unknown', ':', '你', '[SEP]']\n",
            "Encoded: [101, 16285, 131, 100, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            "Segments: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            "Decoded: Unknown : [UNK]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ekc_XyBfELbN"
      },
      "source": [
        "## Vectorize data\n",
        "\n",
        "We'll use the familiar `LabelEncoder` for labels and the keras-bert `Tokenizer` for text data. `Y` is the representation of the labels that will be given to the model for training."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9qFYFHpTELbN",
        "outputId": "08f0f5a1-959f-43a9-ae84-2b394dec49c8"
      },
      "source": [
        "from sklearn.preprocessing import LabelEncoder\n",
        "\n",
        "\n",
        "label_encoder = LabelEncoder()    # Turns class labels into integers\n",
        "Y = label_encoder.fit_transform(labels)\n",
        "\n",
        "# Take note of how many unique labels there are in the data\n",
        "num_labels = len(set(Y))\n",
        "\n",
        "\n",
        "# Print out some examples\n",
        "print('Number of unique labels:', num_labels)\n",
        "print(type(labels), labels[:10])\n",
        "print(type(Y), Y[:10])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of unique labels: 2\n",
            "<class 'list'> ['pos', 'neg', 'pos', 'pos', 'pos', 'neg', 'neg', 'pos', 'neg', 'pos']\n",
            "<class 'numpy.ndarray'> [1 0 1 1 1 0 0 1 0 1]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nja_9VB8ELbO"
      },
      "source": [
        "Keep token indices and segment ids in separate lists and store as numpy arrays. `X` here is the final vectorized form of the input we'll be providing to the model for training. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KBH8wIuZELbO",
        "outputId": "59aa63ff-ffd4-40b5-bac2-e88238bcf150"
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "\n",
        "token_indices, segment_ids = [], []\n",
        "for text in texts:\n",
        "    # tokenizer.encode() returns a sequence of token indices\n",
        "    # and a sequence of segment IDs. BERT expects both as input,\n",
        "    # even if the segments IDs are just all zeros (like here).\n",
        "    tid, sid = tokenizer.encode(text, max_len=INPUT_LENGTH)\n",
        "    token_indices.append(tid)\n",
        "    segment_ids.append(sid)\n",
        "\n",
        "# Format input as list of two numpy arrays\n",
        "X = [np.array(token_indices), np.array(segment_ids)]\n",
        "\n",
        "\n",
        "# Print some examples\n",
        "print('Token indices:')\n",
        "print(X[0][:2])\n",
        "print('Decoded:')\n",
        "for i in X[0][:2]:\n",
        "    print(tokenizer.decode(list(i)))\n",
        "print('Segment ids:')\n",
        "print(X[1][:2])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Token indices:\n",
            "[[  101   146  1486  1130  7301  1306  5813  1665   112   188 26247  1136\n",
            "   1106  1263  2403  1111  1103  1148  1159  1105   146  1138  1106  1474\n",
            "    117   146  1541  1276  1122  1106  1129  2385  1363   119  1409  1128\n",
            "   1132   170  5442  1104 11288 15706 23136  1128  1209  1567  1122   119\n",
            "   1109  4569  2523  2274  1282  1656  1117  1713   118  1137  1674  1122\n",
            "    136  1109  3176  1121  2490  1950  1110   170  1376  6169  1105 16591\n",
            "   1105  1199  1104  1103  4429  1180  1129  2195  1205  1133  1122  1759\n",
            "   1149  1107  1103  1322   119  1109  3908  1116  1113  1103  4173  1132\n",
            "   1198  1112  1632  1112  1103  1273   117  1191  1136  3407  1111  1343\n",
            "  16727  3899   119  1135  1144  5606  1104  1169  3309  1181  4899  1121\n",
            "   1103  1383   117  1149 13482  1116  1105   170  1632  3669  1114  1103\n",
            "   1900   119 15859  1150  1144  2065  1194  1543  1126  2457  1273  1209\n",
            "   1567  1106  2824 16613   113  1103  1900   114   117 16727  1105  2490\n",
            "   1950  1113  1103  1304  1353  1601  2357  1383  2222  1106 21260  1149\n",
            "   1142  1376  3868  5005 19857  1273   119  1135  1108  2785 24815  1105\n",
            "    146   112   182  5171  1106  1138  1122  1107  1139  2436   119   102\n",
            "      0     0     0     0     0     0     0     0     0     0     0     0\n",
            "      0     0     0     0     0     0     0     0     0     0     0     0\n",
            "      0     0     0     0     0     0     0     0     0     0     0     0\n",
            "      0     0     0     0     0     0     0     0     0     0     0     0\n",
            "      0     0     0     0     0     0     0     0     0     0]\n",
            " [  101  1109  1273  1110   170  2436  1104   172 26567   112   188  1113\n",
            "   1198  1164  1625  1149  1175   119  1135  1144  1185  2817 20748   117\n",
            "   1185  2513   117  1185  1842  3802   119   156 17162 15792  1863  1110\n",
            "   2873  1166  1103  1499  1105 17000  2340  2624  1110 13504  1105 22052\n",
            "   2285   119  1188  2523  1169   112   189  9345  1103 23483  1104  1543\n",
            "   3362  1187  1664  5903   119  4081  1353  3670  1104  1734  3587  2411\n",
            "   1154   170 12628   117  8362  9380  2050  6202 17693  8032  1115  1110\n",
            "   1177  4701  1104  1126  3176  1705 20762   119  2777  1175  1110  1185\n",
            "   9556  1106  1142  3670   117  1103  5681   113  8334  1104  1293  1363\n",
            "   1152  1132  5156   114  1169   112   189  1494  1133 20766  1114 18088\n",
            "   7471   117 10908   165  5920  1103 17930   165   107   119  3278  3176\n",
            "    117  1106  1508  1122  2566   119  1109 10404  1110  1104  1185  1494\n",
            "   1303   119  4302  1169  3277  1142  1664   118  1642   119  1135  1110\n",
            "   2257   117  8406  1181  1105 12533  1106  3632   119  1130  7409 10182\n",
            "  19366  1905  1107  4104  4626  9257  7899  2067  1114  1109  2627   117\n",
            "  15659  1116  1105  6746  1183  3078   112   188 24526   119 23543  1116\n",
            "  20004  1443   170  1648   117  6505  1105 17941   119 26316  1116  1132\n",
            "  16439   119  1188  1110 27593 16303 16040  1109  2750 26297  2225 16040\n",
            "   4306  3259 21416   117  1114  2764  1105  1893  8899  1149   119  2750\n",
            "    141  2101   119   146   112  1325  1660  1122  1115   119  1789  2441\n",
            "   1138 24132   119  1188  1273  1110  2499   112   188   102]]\n",
            "Decoded:\n",
            "['I', 'saw', 'In', '##so', '##m', '##nia', '##c', \"'\", 's', 'Nightmare', 'not', 'to', 'long', 'ago', 'for', 'the', 'first', 'time', 'and', 'I', 'have', 'to', 'say', ',', 'I', 'really', 'found', 'it', 'to', 'be', 'quite', 'good', '.', 'If', 'you', 'are', 'a', 'fan', 'of', 'Dominic', 'Mona', '##ghan', 'you', 'will', 'love', 'it', '.', 'The', 'hole', 'movie', 'takes', 'place', 'inside', 'his', 'mind', '-', 'or', 'does', 'it', '?', 'The', 'acting', 'from', 'everyone', 'else', 'is', 'a', 'little', 'rushed', 'and', 'shaky', 'and', 'some', 'of', 'the', 'scenes', 'could', 'be', 'cut', 'down', 'but', 'it', 'works', 'out', 'in', 'the', 'end', '.', 'The', 'extra', '##s', 'on', 'the', 'DVD', 'are', 'just', 'as', 'great', 'as', 'the', 'film', ',', 'if', 'not', 'greater', 'for', 'those', 'Dom', 'fans', '.', 'It', 'has', 'tons', 'of', 'can', '##di', '##d', 'moments', 'from', 'the', 'set', ',', 'out', '##take', '##s', 'and', 'a', 'great', 'interview', 'with', 'the', 'director', '.', 'Anyone', 'who', 'has', 'gone', 'through', 'making', 'an', 'independent', 'film', 'will', 'love', 'to', 'watch', 'Tess', '(', 'the', 'director', ')', ',', 'Dom', 'and', 'everyone', 'else', 'on', 'the', 'very', 'small', 'close', 'personal', 'set', 'try', 'to', 'bang', 'out', 'this', 'little', 'trip', '##py', 'creepy', 'film', '.', 'It', 'was', 'pretty', 'enjoyable', 'and', 'I', \"'\", 'm', 'glad', 'to', 'have', 'it', 'in', 'my', 'collection', '.']\n",
            "['The', 'film', 'is', 'a', 'collection', 'of', 'c', '##liche', \"'\", 's', 'on', 'just', 'about', 'anything', 'out', 'there', '.', 'It', 'has', 'no', 'focus', 'whatsoever', ',', 'no', 'goals', ',', 'no', 'real', 'message', '.', 'S', '##ym', '##bol', '##ism', 'is', 'pushed', 'over', 'the', 'top', 'and', 'stereo', '##ty', '##ping', 'is', 'abundant', 'and', 'outrage', '##ous', '.', 'This', 'movie', 'can', \"'\", 't', 'resist', 'the', 'temptation', 'of', 'making', 'drama', 'where', 'non', 'exists', '.', 'Every', 'small', 'exchange', 'of', 'words', 'turns', 'immediately', 'into', 'a', 'lengthy', ',', 'un', '##ju', '##st', '##ified', 'dial', '##og', 'that', 'is', 'so', 'typical', 'of', 'an', 'acting', 'class', 'rehearsal', '.', 'Where', 'there', 'is', 'no', 'substance', 'to', 'this', 'exchange', ',', 'the', 'actors', '(', 'regardless', 'of', 'how', 'good', 'they', 'are', 'normally', ')', 'can', \"'\", 't', 'help', 'but', 'compensate', 'with', 'exaggerated', 'emotion', ',', 'aka', '\\\\', 'raising', 'the', 'stakes', '\\\\', '\"', '.', 'Over', 'acting', ',', 'to', 'put', 'it', 'simply', '.', 'The', 'directing', 'is', 'of', 'no', 'help', 'here', '.', 'Nothing', 'can', 'save', 'this', 'non', '-', 'story', '.', 'It', 'is', 'forced', ',', 'fake', '##d', 'and', 'boring', 'to', 'tears', '.', 'In', '##ac', '##cu', '##rac', '##ies', 'in', 'port', '##ara', '##ying', 'punk', 'rock', 'with', 'The', 'Who', ',', 'piercing', '##s', 'and', 'flash', '##y', '90', \"'\", 's', 'outfits', '.', 'Character', '##s', 'wander', 'without', 'a', 'role', ',', 'detail', 'and', 'motive', '.', 'Location', '##s', 'are', 'arbitrary', '.', 'This', 'is', 'Boogie', 'Nights', 'cum', 'The', 'Good', 'Fell', '##as', 'cum', 'Saturday', 'Night', 'Fever', ',', 'with', 'meaning', 'and', 'art', 'ripped', 'out', '.', 'Good', 'D', '##P', '.', 'I', \"'\", 'll', 'give', 'it', 'that', '.', 'Some', 'films', 'have', 'flaws', '.', 'This', 'film', 'is', 'Lee', \"'\", 's']\n",
            "Segment ids:\n",
            "[[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
            " [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YApAPEvFELbP"
      },
      "source": [
        "## Load pretrained BERT model\n",
        "\n",
        "We'll use the keras-bert function `load_trained_model_from_checkpoint` to load the model from the checkpoint we downloaded earlier.\n",
        "\n",
        "Explanation for a few parameters from keras-bert documentation:\n",
        "\n",
        "* `training`: If `training`, the whole model will be returned. Otherwise, the MLM [masked language modeling] and NSP [next sentence prediction] parts will be ignored.\n",
        "* `trainable`: Whether the model is trainable. The default value is the same with `training`.\n",
        "\n",
        "We don't need the masked language modeling or next sentence prediction parts (these are primarily for pretraining), so we'll use `training=False` but `trainable=True`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "StLxTwFAELbP",
        "outputId": "fa763c29-6b8e-4055-9086-85fe8891ad52"
      },
      "source": [
        "from keras_bert import load_trained_model_from_checkpoint\n",
        "\n",
        "\n",
        "pretrained_model = load_trained_model_from_checkpoint(\n",
        "    config_file = bert_config_path,\n",
        "    checkpoint_file = bert_checkpoint_path,\n",
        "    training = False,\n",
        "    trainable = True,\n",
        "    seq_len = INPUT_LENGTH\n",
        ")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /Users/smp/Library/Python/3.7/lib/python/site-packages/tensorflow_core/python/keras/initializers.py:119: calling RandomUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
            "WARNING:tensorflow:From /Users/smp/Library/Python/3.7/lib/python/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "If using Keras pass *_constraint arguments to layers.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZdEZUY1CELbQ"
      },
      "source": [
        "Let's have a bit of a look at that model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vl6x4gBoELbQ",
        "outputId": "130f7d71-f87b-40c2-ee83-cef7bdd65e52"
      },
      "source": [
        "# This is a keras model, so we can figure out what inputs it takes like so:\n",
        "pretrained_model.inputs"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<tf.Tensor 'Input-Token:0' shape=(?, 250) dtype=float32>,\n",
              " <tf.Tensor 'Input-Segment:0' shape=(?, 250) dtype=float32>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mi66esRrELbR",
        "outputId": "2f042c5e-90b4-4cb1-bde0-39c0909b5653"
      },
      "source": [
        "# And similarly for outputs:\n",
        "pretrained_model.outputs"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<tf.Tensor 'Encoder-12-FeedForward-Norm/add_1:0' shape=(?, 250, 768) dtype=float32>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "inLvtQsWELbS"
      },
      "source": [
        "So, the model takes `Input-Token` and `Input-Segment` inputs, both of dimension (batch-size, input-length), and produces a single output tensor of dimension (batch-size, input-length, hidden-dim). The input matches our `X`, but we'll need to work on the output a bit as our `Y` is just a label for each input, not a sequence of labels."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w2JnWADCELbS",
        "outputId": "086ad388-8cff-452f-94ca-ebbf7f4f9b27"
      },
      "source": [
        "pretrained_model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_1\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "Input-Token (InputLayer)        [(None, 250)]        0                                            \n",
            "__________________________________________________________________________________________________\n",
            "Input-Segment (InputLayer)      [(None, 250)]        0                                            \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Token (TokenEmbedding [(None, 250, 768), ( 22268928    Input-Token[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Segment (Embedding)   (None, 250, 768)     1536        Input-Segment[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Token-Segment (Add)   (None, 250, 768)     0           Embedding-Token[0][0]            \n",
            "                                                                 Embedding-Segment[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Position (PositionEmb (None, 250, 768)     192000      Embedding-Token-Segment[0][0]    \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Dropout (Dropout)     (None, 250, 768)     0           Embedding-Position[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Norm (LayerNormalizat (None, 250, 768)     1536        Embedding-Dropout[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 250, 768)     2362368     Embedding-Norm[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 250, 768)     0           Encoder-1-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 250, 768)     0           Embedding-Norm[0][0]             \n",
            "                                                                 Encoder-1-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 250, 768)     1536        Encoder-1-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward (FeedForw (None, 250, 768)     4722432     Encoder-1-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward-Dropout ( (None, 250, 768)     0           Encoder-1-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward-Add (Add) (None, 250, 768)     0           Encoder-1-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-1-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward-Norm (Lay (None, 250, 768)     1536        Encoder-1-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 250, 768)     2362368     Encoder-1-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 250, 768)     0           Encoder-2-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 250, 768)     0           Encoder-1-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-2-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 250, 768)     1536        Encoder-2-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward (FeedForw (None, 250, 768)     4722432     Encoder-2-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward-Dropout ( (None, 250, 768)     0           Encoder-2-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward-Add (Add) (None, 250, 768)     0           Encoder-2-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-2-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward-Norm (Lay (None, 250, 768)     1536        Encoder-2-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 250, 768)     2362368     Encoder-2-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 250, 768)     0           Encoder-3-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 250, 768)     0           Encoder-2-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-3-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 250, 768)     1536        Encoder-3-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward (FeedForw (None, 250, 768)     4722432     Encoder-3-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward-Dropout ( (None, 250, 768)     0           Encoder-3-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward-Add (Add) (None, 250, 768)     0           Encoder-3-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-3-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward-Norm (Lay (None, 250, 768)     1536        Encoder-3-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 250, 768)     2362368     Encoder-3-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 250, 768)     0           Encoder-4-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 250, 768)     0           Encoder-3-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-4-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 250, 768)     1536        Encoder-4-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward (FeedForw (None, 250, 768)     4722432     Encoder-4-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward-Dropout ( (None, 250, 768)     0           Encoder-4-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward-Add (Add) (None, 250, 768)     0           Encoder-4-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-4-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward-Norm (Lay (None, 250, 768)     1536        Encoder-4-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 250, 768)     2362368     Encoder-4-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 250, 768)     0           Encoder-5-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 250, 768)     0           Encoder-4-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-5-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 250, 768)     1536        Encoder-5-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward (FeedForw (None, 250, 768)     4722432     Encoder-5-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward-Dropout ( (None, 250, 768)     0           Encoder-5-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward-Add (Add) (None, 250, 768)     0           Encoder-5-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-5-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward-Norm (Lay (None, 250, 768)     1536        Encoder-5-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 250, 768)     2362368     Encoder-5-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 250, 768)     0           Encoder-6-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 250, 768)     0           Encoder-5-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-6-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 250, 768)     1536        Encoder-6-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward (FeedForw (None, 250, 768)     4722432     Encoder-6-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward-Dropout ( (None, 250, 768)     0           Encoder-6-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward-Add (Add) (None, 250, 768)     0           Encoder-6-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-6-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward-Norm (Lay (None, 250, 768)     1536        Encoder-6-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 250, 768)     2362368     Encoder-6-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 250, 768)     0           Encoder-7-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 250, 768)     0           Encoder-6-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-7-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 250, 768)     1536        Encoder-7-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward (FeedForw (None, 250, 768)     4722432     Encoder-7-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward-Dropout ( (None, 250, 768)     0           Encoder-7-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward-Add (Add) (None, 250, 768)     0           Encoder-7-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-7-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward-Norm (Lay (None, 250, 768)     1536        Encoder-7-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 250, 768)     2362368     Encoder-7-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 250, 768)     0           Encoder-8-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 250, 768)     0           Encoder-7-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-8-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 250, 768)     1536        Encoder-8-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward (FeedForw (None, 250, 768)     4722432     Encoder-8-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward-Dropout ( (None, 250, 768)     0           Encoder-8-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward-Add (Add) (None, 250, 768)     0           Encoder-8-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-8-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward-Norm (Lay (None, 250, 768)     1536        Encoder-8-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 250, 768)     2362368     Encoder-8-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 250, 768)     0           Encoder-9-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 250, 768)     0           Encoder-8-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-9-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 250, 768)     1536        Encoder-9-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward (FeedForw (None, 250, 768)     4722432     Encoder-9-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward-Dropout ( (None, 250, 768)     0           Encoder-9-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward-Add (Add) (None, 250, 768)     0           Encoder-9-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-9-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward-Norm (Lay (None, 250, 768)     1536        Encoder-9-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 250, 768)     2362368     Encoder-9-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 250, 768)     0           Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 250, 768)     0           Encoder-9-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 250, 768)     1536        Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward (FeedFor (None, 250, 768)     4722432     Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward-Dropout  (None, 250, 768)     0           Encoder-10-FeedForward[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward-Add (Add (None, 250, 768)     0           Encoder-10-MultiHeadSelfAttention\n",
            "                                                                 Encoder-10-FeedForward-Dropout[0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward-Norm (La (None, 250, 768)     1536        Encoder-10-FeedForward-Add[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 250, 768)     2362368     Encoder-10-FeedForward-Norm[0][0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 250, 768)     0           Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 250, 768)     0           Encoder-10-FeedForward-Norm[0][0]\n",
            "                                                                 Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 250, 768)     1536        Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward (FeedFor (None, 250, 768)     4722432     Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward-Dropout  (None, 250, 768)     0           Encoder-11-FeedForward[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward-Add (Add (None, 250, 768)     0           Encoder-11-MultiHeadSelfAttention\n",
            "                                                                 Encoder-11-FeedForward-Dropout[0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward-Norm (La (None, 250, 768)     1536        Encoder-11-FeedForward-Add[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 250, 768)     2362368     Encoder-11-FeedForward-Norm[0][0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 250, 768)     0           Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 250, 768)     0           Encoder-11-FeedForward-Norm[0][0]\n",
            "                                                                 Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 250, 768)     1536        Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward (FeedFor (None, 250, 768)     4722432     Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward-Dropout  (None, 250, 768)     0           Encoder-12-FeedForward[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward-Add (Add (None, 250, 768)     0           Encoder-12-MultiHeadSelfAttention\n",
            "                                                                 Encoder-12-FeedForward-Dropout[0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward-Norm (La (None, 250, 768)     1536        Encoder-12-FeedForward-Add[0][0] \n",
            "==================================================================================================\n",
            "Total params: 107,518,464\n",
            "Trainable params: 107,518,464\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JCq9YAXuELbT"
      },
      "source": [
        "This is a regular Keras model. In Keras, models behave very much like layers, so we're able to wrap this in our own model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mcEpQlYnELbT"
      },
      "source": [
        "## Build classification model\n",
        "\n",
        "We'll make a *very* simple model for text classification: just attach a dense layer to the output for the special `[CLS]` token, and connect the model inputs to the BERT model inputs.\n",
        "\n",
        "Recall that in the BERT input representation, each sequence starts with the special `[CLS]` token and the corresponding output is used in BERT pretraining for the next sentence prediction task. We'll use the `[CLS]` output similarly for our classification task.\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/TurkuNLP/Deep_Learning_in_LangTech_course/master/bert-representation.png\" style=\"width: 80%\">\n",
        "\n",
        "(Figure from [Devlin et al.](https://arxiv.org/pdf/1810.04805.pdf))\n",
        "\n",
        "First, let's find the output"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fBMD6U3mELbU",
        "outputId": "1d735968-93ea-4f75-9877-51c9d1d4506c"
      },
      "source": [
        "print(pretrained_model.outputs)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[<tf.Tensor 'Encoder-12-FeedForward-Norm/add_1:0' shape=(?, 250, 768) dtype=float32>]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iQ4V6Dd5ELbU"
      },
      "source": [
        "Those dimensions are (minibatch-size, sequence-length, hidden-dim).\n",
        "\n",
        "We'll just need the first sequence position across all elements in the initial (minibatch) dimension:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZEa5TRYJELbV",
        "outputId": "6ef884b0-b6e2-4399-95cf-d6760b8e598d"
      },
      "source": [
        "# model.outputs is a list, here with a single item. Here\n",
        "# pretrained_model.outputs[0] just grabs that item (the output tensor).\n",
        "# Indxing that tensor with [:,0] gives the first position in the sequence\n",
        "# for all elements in the batch (the `:`).\n",
        "bert_out = pretrained_model.outputs[0][:,0]\n",
        "\n",
        "print(bert_out)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Tensor(\"strided_slice:0\", shape=(?, 768), dtype=float32)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sQ7rDu2HELbV"
      },
      "source": [
        "Then we can simply create our model. This is just basic Keras, where the pretrained BERT model is behaving essentially as a layer of our \"wrapping\" model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CY7qVYIUELbV"
      },
      "source": [
        "from tensorflow.keras.models import Model\n",
        "from tensorflow.keras.layers import Dense\n",
        "\n",
        "\n",
        "out = Dense(num_labels, activation='softmax')(bert_out)\n",
        "model = Model(\n",
        "    inputs=pretrained_model.inputs,\n",
        "    outputs=[out]\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DUSwRfIoELbW"
      },
      "source": [
        "## Create optimizer\n",
        "\n",
        "BERT is pretrained with an Adam optimizer with warmup and regularization using weight decay. We won't go into detail on these optimizer settings, but will instead largely copy parameters used in the original BERT work.\n",
        "\n",
        "(If you're interested in tuning the training process, trying different values of `LEARNING_RATE` is a good place to start!)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8bIgesAUELbW"
      },
      "source": [
        "from keras_bert import calc_train_steps, AdamWarmup\n",
        "\n",
        "\n",
        "# Calculate the number of steps for warmup\n",
        "total_steps, warmup_steps = calc_train_steps(\n",
        "    num_example=len(texts),\n",
        "    batch_size=BATCH_SIZE,\n",
        "    epochs=EPOCHS,\n",
        "    warmup_proportion=0.1,\n",
        ")\n",
        "\n",
        "optimizer = AdamWarmup(\n",
        "    total_steps,\n",
        "    warmup_steps,\n",
        "    lr=LEARNING_RATE,\n",
        "    epsilon=1e-6,\n",
        "    weight_decay=0.01,\n",
        "    weight_decay_pattern=['embeddings', 'kernel', 'W1', 'W2', 'Wk', 'Wq', 'Wv', 'Wo']\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AP_wA5jYELbX"
      },
      "source": [
        "## Train model\n",
        "\n",
        "The model is compiled and trained normally. As usual, we'll use `sparse_categorical_crossentropy` loss."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "seJAKbDPELbX"
      },
      "source": [
        "model.compile(\n",
        "    optimizer=optimizer,\n",
        "    loss='sparse_categorical_crossentropy',\n",
        "    metrics=['sparse_categorical_accuracy']\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jp9-bpQeELbX"
      },
      "source": [
        "Training, as usual. (Note: this will take a fair bit of time unless you're running with GPU acceleration.)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aV_4_xIoELbX",
        "outputId": "9021fe14-8be0-4451-de65-4c527181413c"
      },
      "source": [
        "history = model.fit(\n",
        "    X,\n",
        "    Y,\n",
        "    epochs=EPOCHS,\n",
        "    batch_size=BATCH_SIZE,\n",
        "    validation_split=0.1\n",
        ")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /Users/smp/Library/Python/3.7/lib/python/site-packages/tensorflow_core/python/ops/math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
            "Train on 900 samples, validate on 100 samples\n",
            "Epoch 1/3\n",
            "900/900 [==============================] - 2404s 3s/sample - loss: 0.5205 - sparse_categorical_accuracy: 0.7233 - val_loss: 0.2022 - val_sparse_categorical_accuracy: 0.8900\n",
            "Epoch 2/3\n",
            "900/900 [==============================] - 2462s 3s/sample - loss: 0.2061 - sparse_categorical_accuracy: 0.9222 - val_loss: 0.2250 - val_sparse_categorical_accuracy: 0.9100\n",
            "Epoch 3/3\n",
            "900/900 [==============================] - 2231s 2s/sample - loss: 0.0533 - sparse_categorical_accuracy: 0.9867 - val_loss: 0.1078 - val_sparse_categorical_accuracy: 0.9500\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1syXJUIbELbY"
      },
      "source": [
        "Let's plot that training history:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q8VWMUa5ELbY",
        "outputId": "827bf632-c716-40b0-f4bc-21f88fe7f315"
      },
      "source": [
        "%matplotlib inline\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "\n",
        "def plot_history(history):\n",
        "    plt.plot(history.history['sparse_categorical_accuracy'],label=\"Training set accuracy\")\n",
        "    plt.plot(history.history['val_sparse_categorical_accuracy'],label=\"Validation set accuracy\")\n",
        "    plt.legend()\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "plot_history(history)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deVyVZf7/8dfFJi7ILrij5cIiCCJgluZuZfnVzNwy93JaJptxcqZ+U9k040xNyzSO5d6mtjg1tomVto2DirvgggspKsiiKDuHc/3+uA9wQFQ04BwOn+fj4YNz7u18zs3xfW6u+7qvW2mtEUII4bicbF2AEEKI+iVBL4QQDk6CXgghHJwEvRBCODgJeiGEcHAuti6gOj8/Px0UFGTrMoQQolHZuXNnltbav6Z5dhf0QUFBJCYm2roMIYRoVJRSP19pnjTdCCGEg5OgF0IIBydBL4QQDs7u2uhrUlpaSlpaGkVFRbYuRTggd3d3OnTogKurq61LEaJeNIqgT0tLw8PDg6CgIJRSti5HOBCtNdnZ2aSlpdGlSxdblyNEvWgUTTdFRUX4+vpKyIs6p5TC19dX/loUDq1RBD0gIS/qjXy2hKNrFE03QgjhyHLyS/jmYAamMs2k2E51vn0J+lrIzs5myJAhAKSnp+Ps7Iy/v3EB2vbt23Fzc7vmNqZPn86CBQvo0aPHFZdZvHgxXl5eTJ48uW4Kr6XNmzfTokUL4uLiGvR1hWjKzlwoZFNSOvFJGWw7kY1ZQ1QnLwl6W/H19WXPnj0APPfcc7Rq1Yrf/va3VZbRWqO1xsmp5tawVatWXfN1HnnkkV9e7A3YvHkzfn5+Ng/6srIynJ2dbVqDEPXpWGYeGw+ksykpnb1puQB0a9OKX91+MyPDAglt17peXrfRtNHbo6NHjxISEsLkyZMJDQ3l7NmzzJkzh+joaEJDQ1m4cGHFsrfeeit79uzBZDLh5eXFggULiIiIoF+/fpw7dw6AZ555htdee61i+QULFhATE0OPHj3YunUrAPn5+dx7772EhIQwbtw4oqOjK76ErM2fP5+QkBDCw8N56qmnAMjIyGDs2LFER0cTExNDQkICx44dY/ny5bz00kv07t274nXKJSQk0K9fPyIjI+nfvz8pKSkAmEwm5s2bR1hYGOHh4fzrX/8CYNu2bfTr14+IiAhiY2MpKChg+fLlPPHEExXbHDlyJD/99FPFvnjiiScIDw9n+/btPPvss/Tt25ewsDAefvhhyu+AduTIEQYPHkxERARRUVGkpqYyadIkPv/884rt3n///XzxxRe/7JcqRB3SWrM/LZeX4w8z9JXvGfL373kp/jAoxe9G9uDb3wzk6ycH8tsRPQhr71lv54sa3RH9858lkXzmYp1uM6Rda569O/SG1j106BDvvPMO0dHRACxatAgfHx9MJhODBg1i3LhxhISEVFknNzeXgQMHsmjRIp588klWrlzJggULLtu21prt27ezYcMGFi5cyMaNG3njjTcIDAxk/fr17N27l6ioqMvWy8jI4MsvvyQpKQmlFBcuXADg8ccf53e/+x1xcXGkpqYyatQoDhw4wKxZs/Dz86sSxuWCg4P58ccfcXFxYePGjTzzzDN88MEHLFmyhDNnzrB3716cnZ3JycmhqKiICRMmsH79eqKiosjNzaVZs2ZX3X+5ubkMGDCg4guuR48ePP/882itmTRpEhs3buSOO+5g4sSJPPfcc9x9990UFRVhNpuZOXMmS5YsYdSoUZw/f54dO3awZs2a2v3ihKgnZWbNjtQc4pPS2ZSUwekLhTg7KWKCfHggrjPDQwNo69m8QWtqdEFvb2666aaKkAdYu3YtK1aswGQycebMGZKTky8L+ubNm3PHHXcA0KdPH3788ccatz127NiKZVJTUwH46aefKo7QIyIiCA29/AvKx8cHJycnZs+ezV133cWoUaMA+Oabbzh8+HDFcufPn6ewsPCq7+/ChQtMnTqVY8eOVZn+zTff8MQTT1Q0tfj4+LB79246depU8eXj6el51W0DuLm5MWbMmIrn3377LS+99BJFRUVkZWXRp08f4uLiyMrK4u677waMC5wABg8ezKOPPkp2djZr165l/Pjx0vQjbKLYVMZ/j2YRfyCDrw9mkJNfgpuLEwO6+fHE0G4MDQ7Au+W1z+XVl0YX9Dd65F1fWrZsWfE4JSWF119/ne3bt+Pl5cWUKVNq7J9tffLW2dkZk8lU47bLj4avtkxNXF1dSUxM5Ouvv+ajjz5iyZIlbNq0qeIvhNqcPC739NNPM2LECH71q19x9OhRRo4cWet1y7m4uGA2myueW++T5s2bV/y5WlBQwKOPPsquXbto3749zzzzzFX7tyulmDJlCmvWrOHtt9/m/fffv+7ahLhRecUmthw6R3xSOt8dziSv2IRHMxcG9WzDyLBABnb3p2Uz+4hYaaOvQxcvXsTDw4PWrVtz9uxZ4uPj6/w1+vfvz4cffgjA/v37SU5OvmyZS5cucfHiRUaNGsWrr77K7t27ARg6dCiLFy+uWK68bd/Dw4NLly7V+Hq5ubm0b98egNWrV1dMHzZsGG+++SZlZWUA5OTkEBISwsmTJ9m1axdg7I+ysjKCgoLYvXs3WmtSU1PZuXNnja9VWFiIk5MTfn5+XLp0ifXr1wPg7e2Nv78/n332GWB8URQUFABGb6aXXnqJZs2aXbVHkxB1ITuvmA92nGTG6h1EvfA1j63dTcLxbO6OaMuq6X1J/H9D+cfESO7s1dZuQh4a4RG9PYuKiiIkJISePXvSuXNn+vfvX+ev8dhjjzF16lRCQkIq/lVvIsnNzWXs2LEUFxdjNpt55ZVXAKP75ty5c1m1alXFOYTFixczevRo7rvvPv7973+zePFibrnlloptPfXUU8yYMYPnn3++orkJ4KGHHiIlJYXw8HBcXFyYO3cuDz/8MGvXrmXu3LkUFRXRvHlzNm/ezMCBA2nfvj3BwcGEhobSu3fvGt+br68vDz74ICEhIbRt25bY2NiKee+//z4PPfQQTz/9NG5ubqxfv57OnTvTrl07unfvzoQJE+pyNwtR4fSFQuIPpBOflM6O1BzMGjp4N+eBuM6MCA2kT2dvnJ3s+6I7Vd6rwV5ER0fr6jceOXjwIMHBwTaqyL6YTCZMJhPu7u6kpKQwfPhwUlJScHFpmt/Z+fn59OrVi7179+Lh4XHD25HPmLB29Nwl4pMy2Hggnf2njW6Q3QNaMTI0kOGhRjdIe7uiWim1U2sdXdO8ppkOjVheXh5DhgzBZDKhteatt95qsiEfHx/P7NmzmT9//i8KeSG01uxLyyU+KZ2NSekcz8wHoHdHLxbc0ZMRoYF08Wt5ja3Yr6aZEI2Yl5fXFdu4m5oRI0Zw8uRJW5chGilTmZntqTlsSsogPimds7lFODsp4rr6MO2WIIaHBBLo6W7rMuuEBL0QoskoKjW6QW48kM43BzM4X1BKMxcnBnT35zfDezA0uA1eLWzXDbK+SNALIRzapaJSthzOJP5AOt8dPkd+SRkezVwYEtyGEaGBDOzhTws3x45Cx353QogmKSuvmG+SM9iYlM7Wo9mUlJnxa9WM0ZHtGREaSL+uvri5NJ3e5RL0QgiHkHa+gPikDOIPpJP4s9ENsqNPc6b268zIsEAiO9l/N8j60nS+0n6BQYMGXXbx02uvvcbcuXOvul6rVq0AOHPmDOPGjatxmdtvv53q3Umre+211youEAK48847K8avaSipqakyjoywK1prUjIu8ca3KYx640du/esWXvg8mYtFpTw6uBtfPn4bP8wfxDOjQogO8mmyIQ9yRF8rEydOZN26dYwYMaJi2rp16/jb3/5Wq/XbtWvHxx9/fMOv/9prrzFlyhRatGgBwJdffnnD27pR5UE/adKkBn9tayaTqcl2JxVgNmv2pl0gPimDTUnpHM8yukFGdfLi95ZukEGNuBtkfZEj+loYN24cX3zxBSUlJYARemfOnOG2226r6NceFRVFr169+M9//nPZ+qmpqYSFhQHGZf4TJkwgODiYMWPGVBlUbO7cuRVDHD/77LMA/OMf/+DMmTMMGjSIQYMGARAUFERWVhYAr7zyCmFhYYSFhVWMAJmamkpwcDCzZ88mNDSU4cOH1zh42UcffURYWBgREREMGDAAMMaEnz9/Pn379iU8PJy33noLgAULFvDjjz/Su3dvXn311Srbudo+eOeddwgPDyciIoIHHngAMEbXHDNmDBEREURERLB169Yq+wjg5Zdf5rnnngOMv3qeeOIJoqOjef311/nss8+IjY0lMjKSoUOHkpGRUVHH9OnT6dWrF+Hh4axfv56VK1dWGZVz2bJlzJs37yq/bWFvTGVmth7N4o//OcAtizYz5l9bWf7jcdp7N+eF/wtj2x+G8O9f9eehgTdJyF9B4zs0+moBpO+v220G9oI7Fl1xto+PDzExMXz11VeMHj2adevWMX78eJRSuLu788knn9C6dWuysrKIi4vjnnvuueJVc0uWLKFFixYcPHiQffv2VRlm+MUXX8THx4eysjKGDBnCvn37ePzxx3nllVfYsmULfn5+Vba1c+dOVq1axbZt29BaExsby8CBA/H29iYlJYW1a9eybNkyxo8fz/r165kyZUqV9RcuXEh8fDzt27evaApasWIFnp6e7Nixg+LiYvr378/w4cNZtGgRL7/8cpXx38tdaR8kJyfzpz/9ia1bt+Ln50dOTg5gDJc8cOBAPvnkE8rKysjLy+P8+fNX/RWVlJRUNHGdP3+ehIQElFIsX76cv/3tb/z973/nhRdewNPTk/3791cs5+rqyosvvshLL72Eq6srq1atqvjyEvarqLSMH1OyiE8yukFeKCjF3dWJgd39+V1oD4b0DMCzhauty2w0Gl/Q20h580150K9YsQIw2gn/8Ic/8MMPP+Dk5MTp06fJyMggMDCwxu388MMPPP744wCEh4cTHh5eMe/DDz9k6dKlmEwmzp49S3JycpX51f3000+MGTOmYgTNsWPH8uOPP3LPPffQpUuXijFlrIc5tta/f3+mTZvG+PHjK4ZE3rRpE/v27atoasrNzSUlJeWqI15eaR9s3ryZ++67r+ILysfHBzDuaPXOO+8Axsicnp6e1wz6+++/v+JxWloa999/P2fPnqWkpIQuXboAxtDJ69atq1jO29sbMIYz/vzzzwkODqa0tJRevXpd9bWEbVwsKq0yGmRBSRmt3V0YEhxgdIPs7k9zNxmG+kY0vqC/ypF3fRo9ejTz5s1j165dFBQU0KdPH8AYbCszM5OdO3fi6upKUFDQVYfWvZITJ07w8ssvs2PHDry9vZk2bdoNbaec9Q0/nJ2da2y6efPNN9m2bRtffPEFffr0YefOnWiteeONN6qcjwD47rvvrvhadbEPrjaUMVQdDvqxxx7jySef5J577uG7776raOK5klmzZvHnP/+Znj17Mn369OuqS9SvzEvFfJ1sXJm69VgWpWUaf49mjIlsz8iwQOK6+uLqLC3Mv5TswVpq1aoVgwYNYsaMGUycOLFiem5uLm3atMHV1ZUtW7bw888/X3U7AwYMqOi9cuDAAfbt2wcYQ/q2bNkST09PMjIy+OqrryrWudIwwrfddhuffvopBQUF5Ofn88knn3DbbbfV+j0dO3aM2NhYFi5ciL+/P6dOnWLEiBEsWbKE0tJSwLiFX35+/jWHMq5pHwwePJiPPvqI7OxsgIqmmyFDhrBkyRLAOCeQm5tLQEAA586dIzs7m+Li4hqbiKxfr3zo5Lfffrti+rBhw6oMw1z+V0JsbCynTp1izZo1VX53wjZO5RSw/Mfj3PfmVmL+/A1/+GQ/qdn5TO/fhfVzb2Hb74fw4phe3NbNX0K+jjS+I3obmjhxImPGjKnSPDB58mTuvvtuevXqRXR0ND179rzqNubOncv06dMJDg4mODi44i+DiIgIIiMj6dmzJx07dqwyxPGcOXMYOXIk7dq1Y8uWLRXTo6KimDZtGjExMYBx5BoZGVljM01N5s+fT0pKClprhgwZQkREBOHh4aSmphIVFYXWGn9/fz799FPCw8NxdnYmIiKCadOmVTmheaV9EBoaytNPP83AgQNxdnYmMjKS1atX8/rrrzNnzhxWrFiBs7MzS5YsoV+/fvzxj38kJiaG9u3bX3U/Pvfcc9x33314e3szePBgTpw4ARj33H3kkUcICwvD2dmZZ599tqJJavz48ezZs6eiOUc0HK01RzLyjAHDDqSTfNa4FWhw29b8ekg3RoQG0jPQw+5Gg3QkMkyxaBJGjRrFvHnzGDJkSI3z5TNWt8xmzZ60C8QnpRN/IJ3U7AKUgqhO3owMDWREaCCdfFvYukyHIsMUiybrwoULxMTEEBERccWQF3WjtMzMtuOWm2Inp5NxsRgXJ0W/m3yZPaArw0ICaOPhGKNBNja1Cnql1EjgdcAZWK61XlRtfmdgJeAP5ABTtNZplnllQHl/yJNa63vqqHYhrsnLy4sjR47YugyHVVhSxg8pmcQnpfPtwXPkFpbS3NWZgd39GRkWyKCebfBsLt0gbe2aQa+UcgYWA8OANGCHUmqD1tr6ZqUvA+9ord9WSg0G/gI8YJlXqLWu+d5x10FrLW14ol7YW/OlvcstLGXzoQziD2Tw/ZFMCkvL8GzuWjEa5IBu0g3yhpUWgmvzOt9sbY7oY4CjWuvjAEqpdcBowDroQ4AnLY+3AJ/WZZHu7u5kZ2fj6+srYS/qlNaa7Oxs3N2lSeFqzl0qqrhBx/+OZWMyawJaN2Ncnw6MCA0ktquP9JC5HlrDpbNwZg+c3QtnLT99b4ZpV+5xdqNqE/TtgVNWz9OA2GrL7AXGYjTvjAE8lFK+WutswF0plQiYgEVa68u+BJRSc4A5AJ06dbqsgA4dOpCWlkZmZmYtyhXi+ri7u9OhQwdbl2F3TmYXVNxab9fJ82gNQb4tmHlbF0aEBtK7gxdOTXigsFrTGnJPGUFuHez55XmmwK87dBkAnfrVSwl1dTL2t8A/lVLTgB+A00CZZV5nrfVppVRXYLNSar/W+pj1ylrrpcBSMHrdVN+4q6trxdWPQoj6obXmUPqlim6Qh9KN6yZC2rZm3tDujAgNpHtAK/mr+mq0hvOplUfo5cFeaFxDgnIG/55w8zBo1xvaRhhDsLjV7xg9tQn600BHq+cdLNMqaK3PYBzRo5RqBdyrtb5gmXfa8vO4Uuo7IBKoEvRCCNswmzW7T50nPimDjQfSOZljdIOM7uzNM3cFMyI0kI4+0g2yRmYznD8BZ3ZXBvvZvVCUa8x3coE2wdDzLiPQ20VCQGi9tMFfS22CfgfQTSnVBSPgJwBVxqpVSvkBOVprM/B7jB44KKW8gQKtdbFlmf5A7cb2FULUi9IyMwnHs9l4IJ2vkzM4d6kYV2fFLTf58fDAmxgWEoC/R7Nrb6gpMZdB9tGqR+np+6DYuPgLZzcjxEPHQFvLkXpAKLjYx368ZtBrrU1KqUeBeIzulSu11klKqYVAotZ6A3A78BellMZounnEsnow8JZSyowx3MKiar11hBANoLCkjO+PlHeDzOBikYnmrs4M6unPiFCjG2Rrd+kGCUCZCbKOVG1+Sd8PpcbY97i4Q0AYhI83Ar1tb6M5xsV+byreKK6MFUJcv9yCUr49ZDTJ/JCSSVGpGa8Wrgy1jAZ5Wzc/3F2beDfIslI4d7Bqz5f0A2CyDALo2gICwy1NL5Yjdb8e4Gx/15rKlbFCNBHnLhYRn2zcNzXhuNENMrC1O+OjOzIyNJCYLj64NNVukKZiOJdctedLRhKUGTcUws0D2oZD9IzKYPe9GZwa/5ehBL0QjVxqVr4xpkxSOrtOGjeQ6erXklm3dWVkWCDh7T2bXjfI0kIjxM/uqQz2cwfBbIzKirunEeaxD1na1HuDT1dwcswvQQl6IRoZrTXJZy9W3De1vBtkWPvW/GZYd0aGBXJzmybUDbIk32husW5TzzwE2tLDu7m3EeT9HqlsfvHuAk1l/yBBL0SjYDZrdp08z8YD6cQnp3MqpxCloG+QD/9vVAjDQwKaRjfI4ktwdl/VNvWsI6AtN61p4WeEeY87KptfPDs2qVCviQS9EHaqxGTmf8ezjdEgkzLIyivGzdmJ/jf78sjtNzM0JAC/VvbRfa9eFF4wujBat6lnHwMsHUhaBRpBHjK6sktj63ZNPtRrIkEvhB0pKDHx/WFLN8hD57hUZKKFmzODerRhRFggg3r44+GI3SALci6/mvT8icr5rTsYQR5+v6VLYwR41HxfZnE5CXohbOxCQQnfHDRuiv3DkUyKTWa8W7gyMjSQkWGB9L/ZwbpB5mdZwny3Jdj3Qu7JyvlenYwj9MgpxhF7YAS08rddvQ5Agl4IG0jPLWJTstFTJuF4DmVmTVtPdybGdGJEaCB9g7wdoxvkpfTLB/O6aDWCik9X6NAH+s6sPFJv4WO7eh2UBL0QDeSEpRvkxgPp7Dll6Qbp35KHBnRlRGgg4R08G29PGa3h4plqzS97IC/DsoAy+qR3vqXyatLAXtDcy6ZlNxUS9ELUE601SWcussky1O+RjDwAwjt4Mn9ED0aEBnBzGw8bV3kDtIYLJ6v2fDmzBwqyjPnKybh6tOugqiM0NmuE79VBSNALUYfKzJqdP5+vuIAp7XwhTpZukM/eHcLw0EDaezX86IU3TGvLCI3VbpBReN6Yr5yNERq7j6zszhgQWu/D7orrI0EvxC9UbCpj67FsNiUZo0Fm5ZXg5uzErd38eHxwN4YEt8G3MXSDNJsh55jlCN1yovTsPiguH3bXFQJCIPjuyqtJA0JsMuyuuD4S9ELcgPxiE99ZukFuOXSOS8UmWro5M6incd/U2+29G6S5DLJSqo3QuA9KjOYlnJsZR+ZhYy3NL72NI3c7GXZXXB8JeiFqKbegtKKnzA8pWZSYzPi0dOPOXm0ZERbALTfZaTfIMpMxJECVERr3Q2mBMd+lOQSGQcTEyuYX/57gbMdfVOK6SNALUQtJZ3J5YMV2cvJLaOfpzuRYoxtkdGc76wZpKoHMg5eP0GgqMua7tjRGaIyaWnk1qV93uxx2V9Qd+e0KcQ3703KZsmIbLd2cWT/3FqI6edlHN8jSImPY3SojNCZXG3Y3AvrOquzS6HuTQwy7K66PBL0QV7Hn1AUeWLGN1u6urJsTZ7uBw0oLrUZotB5212TMd/c0gjz24co2de8uDjvsrrg+EvRCXMHOn88zbeV2vFu6sWZ2LB28Gyjki/Mg40DV5pfMw1bD7voYYX7LsMo2da/OMpiXuCIJeiFqsP1EDtNXbcffoxlr58TR1rOeuhAWXTR6u1gPE5B1hIoRGlu2MYK8512VbeqeHSTUxXWRoBeimv8dy2bG6h209XJn7ew4Alq7182GC89bxlK36tKYc6xyvkdbI8zDxla2qXsESqiLX0yCXggrP6VkMeudHXT0bsH7s2Np43GDIZ+fXRno5T/Pp1bO9+xohHl5l8a2EeARUCfvQYjqJOiFsPj+SCZz3kmki19L3psVW/ubeuSdqzqQ19m9kHuqcr5XZ6P5xbpLY0u/+nkTQtRAgl4IYPOhDB5+dxc3t2nFe7Ni8WnpdvlCWluG3a12g4xLZyqX8bkJOvSFmNmVR+rNvRvujQhRAwl60eR9nZzBb97/L7f7m/j7yBZ4nPjcCPS8dLiUYfmZDpfOQpFl3BcU+HWDoFsre74E9jK6OQphZyTohePS2riZdF6GEdJVQjsd8jLIy04j7mI6+1wL4QKw1mp9ZzfjvqQeAcZY6kG3gm83q2F3W9nqnQlxXSToReOjNRRdqBLYlY+rHYWXj+dizaU5eASQ4+TD/y60QbfqxdCYCNy92xm9XFoFGj+be0uPF+EQJOiF/TCboTDHKrCvEuRlxZev79aqMqjbRVYGtkcgtAqofNysNf/Ze4Z5H+whurMPK6f3xb2Z/FcQjks+3aL+mcuMG0JfOlsZ2pc1p1h+ll/Sb83ds7IJpWNcDeHd1nhcy6aU9TvTmP/xXmK6+LByWl9auMl/A+HYavUJV0qNBF4HnIHlWutF1eZ3BlYC/kAOMEVrnWaZ9yDwjGXRP2mt366j2oWtlZUaXQvLg/pKQZ5/DrT58vWb+1SGtl8PI8hrOgqvwxtbfLjjFE/9ex+33OTL8ql9ae4mA3wJx3fNoFdKOQOLgWFAGrBDKbVBa51stdjLwDta67eVUoOBvwAPKKV8gGeBaIxrunda1j1f129E1CFTcbXmkiu0gednUXGpfgVl9BEvb0IJ7FUtvMt/tmnwm1i8v+1nnv7kAAO6+7P0gT72OXa8EPWgNkf0McBRrfVxAKXUOmA0YB30IcCTlsdbgE8tj0cAX2utcyzrfg2MpGrfBtFQSgpq6DJYQ5AX1vA9rJyNcG4VAJ7toUOfyuYU6yBv6W+XN6x453+p/PE/SQzu2YZ/TY6SkBdNSm2Cvj1gdZkfaUBstWX2AmMxmnfGAB5KKd8rrNu++gsopeYAcwA6depU29pFueJLNTSdVA/yjMp7f1pzcrU0kQQYY5V3vqXa0bclyFv6NdpxzFf8dIIXPk9mWEgA/5wUSTOXxvk+hLhRdXUW6rfAP5VS04AfgNNAWW1X1lovBZYCREdHV28LaJoquhBmXLsNvDT/8vWdm1UGtn9P6Hr75eHt0dboQujAY5a/9f0x/vLVIUaGBvKPiZG4uTjuexXiSmoT9KeBjlbPO1imVdBan8E4okcp1Qq4V2t9QSl1Gri92rrf/YJ6Gz+toSDHEt5nrx7k5bd/s+basjKo2/aG7tW6DpYHubtXk+8DvnjLUV6KP8xd4W157f7euNrTLf+EaEC1CfodQDelVBeMgJ8ATLJeQCnlB+Rorc3A7zF64ADEA39WSpUP9jHcMt/xmM1QkHX18C4PcHPp5es3a10Z2B1jLu86WNEH3KPh31sj9Po3Kbz6zRFG927H3++LsK/7ugrRwK4Z9Fprk1LqUYzQdgZWaq2TlFILgUSt9QaMo/a/KKU0RtPNI5Z1c5RSL2B8WQAsLD8x22iUmYzugVfq+10e5HnnKu8AZK25t9Vl9LfWfAFPq0Bws9Et6hyM1ppXvj7CG5uPcm9UBzoD0AAAABe1SURBVP42Lhxnp6b9l40QSmv7ahKPjo7WiYmJ9f9CppLLT1zWFOT5mVzehRBo4WcV2G1r7gPeKgBc6+imFeKatNb8deNh3vz+GPdHd+QvY3vhJCEvmgil1E6tdXRN8xzvksDSwmuPf3Ip3bjUvjrlZHQP9AgEj3Y1XEZvOTJv2QZcahjGVtiM1poXvzjI8p9OMDm2Ey+MDpOQF8LCcYL+UjosjrEaRtaKk0vlEbZ3EHSKq7kPeAs/cHacXdJUaK15/rNkVm9N5cF+nXnunlBUEz8RLYQ1x0m15j7Qa3zVroPlj1v4OnQXwqbMbNb8ccMB3ks4ycxbu/DMXcES8kJU4zhB7+IGd71s6ypEAzKbNU9/up+120/x0MCuLBjZU0JeiBo4TtCLJqXMrFmwfh8f7Uzj0UE385vh3SXkhbgCCXrR6JSZNfM/2su/d5/miaHd+PWQbhLyQlyFBL1oVExlZuZ9uJfP9p7ht8O78+jgbrYuSQi7J0EvGo3SMjO/XrebL/ens+COnjw88CZblyREoyBBLxqFEpOZx9buIj4pg2fuCmbWbV1tXZIQjYYEvbB7xaYyHnl/F98cPMdzd4cwrX8XW5ckRKMiQS/sWlFpGQ+/t5PvDmfywv+F8UBcZ1uXJESjI0Ev7FZhSRlz3k3kp6NZ/GVsLybGyE1phLgREvTCLhWUmJi5OpGEE9n87d5w7ovueO2VhBA1kqAXdiev2MSM1TtITM3hlfERjInsYOuShGjUJOiFXblUVMq0VTvYc+oCr02I5J6IdrYuSYhGT4Je2I3cwlIeXLmdA6dzeWNiJHf2amvrkoRwCBL0wi5cKChh6srtHDx7kcWToxgRGmjrkoRwGBL0wubO55cwefk2jp7L480pfRgSHGDrkoRwKBL0wqay84qZvHwbx7PyWTq1D7f3aGPrkoRwOBL0wmYyLxUzeXkCP2cXsOLBaG7r5m/rkoRwSBL0wibOXSxi4rIEzlwoYtX0vtxyk5+tSxLCYUnQiwaXnlvEpGUJpF8sYvX0vsR29bV1SUI4NAl60aBOXyhk0rIEsvNKeHdmDH06+9i6JCEcngS9aDCncgqYuCyB3MJS3p0ZQ2Qnb1uXJESTIEEvGsTP2flMWraNvGIT78+KJbyDl61LEqLJkKAX9e5EVj4TlyZQbCrj/VmxhLX3tHVJQjQpEvSiXh09l8ekZQmYzJo1s+MIbtva1iUJ0eRI0It6cyTjEpOWbQNg3Zw4ugd42LgiIZomJ1sXIBzTwbMXmbg0ASclIS+ErdUq6JVSI5VSh5VSR5VSC2qY30kptUUptVsptU8pdadlepBSqlAptcfy7826fgPC/iSdyWXSsgRcnZ344KF+3Nymla1LEqJJu2bTjVLKGVgMDAPSgB1KqQ1a62SrxZ4BPtRaL1FKhQBfAkGWece01r3rtmxhr/an5TJlxTZaujmzdk4cnX1b2rokIZq82hzRxwBHtdbHtdYlwDpgdLVlNFB+ls0TOFN3JYrGYvfJ80xanoCHuwsfPNRPQl4IO1GboG8PnLJ6nmaZZu05YIpSKg3jaP4xq3ldLE063yulbqvpBZRSc5RSiUqpxMzMzNpXL+zGzp9zeGDFdrxbuPHBQ/3o6NPC1iUJISzq6mTsRGC11roDcCfwrlLKCTgLdNJaRwJPAmuUUpf1r9NaL9VaR2uto/39ZQTDxmb7iRymrtiOv0czPngojvZezW1dkhDCSm2C/jTQ0ep5B8s0azOBDwG01v8D3AE/rXWx1jrbMn0ncAzo/kuLFvZj67EsHly5nUBPd9bNiaOtp4S8EPamNkG/A+imlOqilHIDJgAbqi1zEhgCoJQKxgj6TKWUv+VkLkqprkA34HhdFS9s66eULGas3kEH7+asm9OPgNbuti5JCFGDa/a60VqblFKPAvGAM7BSa52klFoIJGqtNwC/AZYppeZhnJidprXWSqkBwEKlVClgBh7WWufU27sRDea7w+eY8+5Ouvq15P1Zsfi2ambrkoQQV6C01rauoYro6GidmJho6zLEVWw+lMHD7+7i5jateG9WLD4t3WxdkhBNnlJqp9Y6uqZ5MgSCuC6bktJ5ZM0uega25t2ZMXi1kJAXwt5J0Ita+2r/WR5bu5uw9p68PSMGz+auti5JCFELEvSiVj7be4YnPthD745erJ7eFw93CXkhGgsZ1Exc06e7T/Prdbvp08mbt2fESMgL0cjIEb24qo93pjH/473EdfFlxbRoWrjJR0aIxkb+14or+mDHSRb8ez+33uzH0geiae7mbOuShBA3QJpuRI3eS/iZp9bvZ0A3f5ZNlZAXojGTI3pxmbe3pvLshiSG9GzDv6ZE0cxFQl6IxkyCXlSx/Mfj/OmLgwwPCeCfk6Jwc5E/+oRo7CToRYU3vz/Goq8OcWevQF6fEImrs4S8EI5Agl4A8M/NKby86Qh3R7Tj1fERuEjIC+EwJOibOK01r3+bwmvfpDAmsj0vjQuXkBfCwUjQN2Faa/6+6Qj/3HKUcX068Nd7w3F2UrYuSwhRxyTomyitNYs2HuKt748zoW9H/jymF04S8kI4JAn6JkhrzYtfHGT5TyeYEteJhfeEScgL4cAk6JsYrTXPf5bM6q2pTLsliGfvDkEpCXkhHJkEfRNiNmv+uOEA7yWcZNatXXj6rmAJeSGaAAn6JsJs1vzhk/2s23GKhwfexFMje0jIC9FESNA3AWVmzVPr9/HxzjQeG3wzTw7rLiEvRBMiQe/gTGVm5n+8j092n2be0O78emg3W5ckhGhgEvQOzFRmZt6He/ls7xnmj+jBI4NutnVJQggbkKB3UKVlZh5fu5uvDqTz+zt68tDAm2xdkhDCRiToHVCJycyja3axKTmDZ+4KZtZtXW1dkhDChiToHUyxqYxfvbeLbw+d4/l7QnnwliBblySEsDEJegdSVFrGQ+/u5Psjmbw4JozJsZ1tXZIQwg5I0DuIwpIyZr+TyH+PZfHXe3txf99Oti5JCGEnJOgdQEGJiZmrE0k4kc1L4yIY16eDrUsSQtgRCfpGLq/YxIxVO0j8OYfX7u/N6N7tbV2SEMLO1OoOE0qpkUqpw0qpo0qpBTXM76SU2qKU2q2U2qeUutNq3u8t6x1WSo2oy+KbuktFpTy4cjs7T57nHxMjJeSFEDW65hG9UsoZWAwMA9KAHUqpDVrrZKvFngE+1FovUUqFAF8CQZbHE4BQoB3wjVKqu9a6rK7fSFOTW1jK1JXbSTqdyz8nRnJHr7a2LkkIYadqc0QfAxzVWh/XWpcA64DR1ZbRQGvLY0/gjOXxaGCd1rpYa30COGrZnvgFLhSUMGX5NpLP5PKvyVES8kKIq6pN0LcHTlk9T7NMs/YcMEUplYZxNP/YdayLUmqOUipRKZWYmZlZy9KbpvP5JUxato3D6Zd464E+DA8NtHVJQgg7V1d3gZ4IrNZadwDuBN5VStV621rrpVrraK11tL+/fx2V5Hiy84qZuCyBo5l5LHswmsE9A2xdkhCiEahNr5vTQEer5x0s06zNBEYCaK3/p5RyB/xqua6ohcxLxUxensDJnAJWPtiXW7v52bokIUQjUZuj7h1AN6VUF6WUG8bJ1Q3VljkJDAFQSgUD7kCmZbkJSqlmSqkuQDdge10V31Scu1jEhKX/41ROIaumxUjICyGuyzWP6LXWJqXUo0A84Ays1FonKaUWAola6w3Ab4BlSql5GCdmp2mtNZCklPoQSAZMwCPS4+b6nM0tZNKybZy7WMTbM2KI6eJj65KEEI2MMvLYfkRHR+vExERbl2EXTl8oZOLSBHLyS3h7Rl/6dJaQF0LUTCm1U2sdXdM8uTLWTp3KKWDisgRyC0t5b1YsvTt62bokIUQjJUFvh37Ozmfi0gTyS8pYMyuOXh08bV2SEKIRk6C3M8cz85i0bBvFpjLWzI4ltJ2EvBDil5GgtyNHz+UxaVkCZWbN2jlx9Axsfe2VhBDiGiTo7cSRjEtMWpYAKNbNiaNbgIetSxJCOIi6ujJW/AIHz15kwtIEnJSEvBCi7skRvY0dOJ3LlBXbaO7qzJrZcXTxa2nrkoQQDkaO6G1oX9oFJi1LoKWbCx/M6SchL4SoF3JEbyO7T55n6srteLVwZc2sODr6tLB1SUIIByVBbwOJqTlMW7UD31ZurJ0dRzuv5rYuSQjhwKTppoFtO57N1JXbaePRjA/m9JOQF0LUOzmib0Bbj2Uxc3Ui7b2bs2ZWLG1au9u6JCFEEyBH9A3kp5QsZqzeQSefFqydHSchL4RoMBL0DeC7w+eY8fYOgnxbsmZ2LP4ezWxdkhCiCZGmm3r27cEM5r63i24BrXhvZizeLd1sXZIQoomRoK9H8UnpPLpmF8FtW/PujFg8W7jauiQhRBMkTTf15Mv9Z3nk/V2EtvPk3ZkS8kII25Ej+nrw2d4zPPHBHiI7erFqel883CXkhRC2I0Ffxz7ZncZvPtxLdJAPq6b1pWUz2cVCCNuSpps69FHiKZ78cC9xXX1ZPV1CXghhHySJ6si67Sf5/Sf7ufVmP5Y+EE1zN2dblySEEIAc0deJdxN+ZsG/9zOwuz/LpkrICyHsixzR/0Kr/3uC5z5LZmhwGxZPjqKZi4S8EMK+SND/Ast/PM6fvjjIiNAA3pgYhZuL/IEkhLA/EvQ3aMl3x/jrxkPc1astr03ojauzhLwQwj5J0N+AN75N4e9fH+GeiHa8Mj4CFwl5IYQdk6C/DlprXvsmhde/TWFsZHteui8CZydl67KEEOKqJOhrSWvN3zcd4Z9bjnJfnw4sujdcQl4I0ShI0NeC1ppFGw/x1vfHmRjTiRf/LwwnCXkhRCNRq8ZlpdRIpdRhpdRRpdSCGua/qpTaY/l3RCl1wWpemdW8DXVZfEPQWvOnLw7y1vfHeSCus4S8EKLRueYRvVLKGVgMDAPSgB1KqQ1a6+TyZbTW86yWfwyItNpEoda6d92V3HC01jz/WTKrt6YyvX8QfxwVglIS8kKIxqU2R/QxwFGt9XGtdQmwDhh9leUnAmvrojhbMps1z3x6gNVbU5kzoKuEvBCi0apN0LcHTlk9T7NMu4xSqjPQBdhsNdldKZWolEpQSv3fFdabY1kmMTMzs5al1x+zWfOHT/bz/raT/Or2m/j9HT0l5IUQjVZdn4ydAHystS6zmtZZa31aKdUV2KyU2q+1Pma9ktZ6KbAUIDo6WtdxTdelzKz53cf7WL8rjceHdGPe0G4S8kKIRq02R/SngY5WzztYptVkAtWabbTWpy0/jwPfUbX93q6Yysz85sM9rN+VxpPDuvPksO4S8kKIRq82Qb8D6KaU6qKUcsMI88t6zyilegLewP+spnkrpZpZHvsB/YHk6uvag9IyM098sIdP95xh/ogePD6km61LEkKIOnHNphuttUkp9SgQDzgDK7XWSUqphUCi1ro89CcA67TW1k0vwcBbSikzxpfKIuveOvaixGTm1+t289WBdP5wZ0/mDLjJ1iUJIUSdUVVz2faio6N1YmJig71esamMR9fs5uvkDP7fqBBm3tqlwV5bCCHqilJqp9Y6uqZ5TfrK2KLSMn71/i42HzrHwtGhTO0XZOuShBCizjXZoC8qLWPOuzv54Ugmfx7Ti0mxnWxdkhBC1IsmGfSFJWXMfieR/x7L4m/3hjO+b8drrySEEI1Ukwv6/GITM9/ewfYTObw8LoJ7+3SwdUlCCFGvmlTQ5xWbmL5qOzt/Ps+r9/dmdO8aL/AVQgiH0mSC/lJRKQ+u3M7etFzemBjFXeFtbV2SEEI0iCYR9LmFpUxduZ2k07ksnhTJyDAJeSFE0+HwQX+hoIQHVmznUPpFlkzpw7CQAFuXJIQQDcqhgz4nv4Qpy7dxNDOPpQ9EM6hnG1uXJIQQDc5hgz4rr5gpy7dxIiuf5VOjGdDd39YlCSGETThk0J+7VMTkZds4db6AldP60v9mP1uXJIQQNuNwQZ9xsYiJyxJIzy1i9fQY4rr62rokIYSwKYcK+rO5hUxato1zF4t4e0YMfYN8bF2SEELYnMME/dncQu5/K4Hz+SW8OyuWqE7eti5JCCHsgsMEfWt3V7q1acXjQ7oR0dHL1uUIIYTdcJigb9nMhRXT+tq6DCGEsDu1uZWgEEKIRkyCXgghHJwEvRBCODgJeiGEcHAS9EII4eAk6IUQwsFJ0AshhIOToBdCCAentNa2rqEKpVQm8PMv2IQfkFVH5dQlqev6SF3XR+q6Po5YV2etdY3jsdtd0P9SSqlErXW0reuoTuq6PlLX9ZG6rk9Tq0uaboQQwsFJ0AshhINzxKBfausCrkDquj5S1/WRuq5Pk6rL4drohRBCVOWIR/RCCCGsSNALIYSDazRBr5QaqZQ6rJQ6qpRaUMP8ZkqpDyzztymlgqzm/d4y/bBSakQD1/WkUipZKbVPKfWtUqqz1bwypdQey78NDVzXNKVUptXrz7Ka96BSKsXy78EGrutVq5qOKKUuWM2rz/21Uil1Til14ArzlVLqH5a69ymloqzm1ef+ulZdky317FdKbVVKRVjNS7VM36OUSmzgum5XSuVa/b7+aDXvqp+Beq5rvlVNByyfKR/LvPrcXx2VUlssWZCklPp1DcvU32dMa233/wBn4BjQFXAD9gIh1Zb5FfCm5fEE4APL4xDL8s2ALpbtODdgXYOAFpbHc8vrsjzPs+H+mgb8s4Z1fYDjlp/elsfeDVVXteUfA1bW9/6ybHsAEAUcuML8O4GvAAXEAdvqe3/Vsq5byl8PuKO8LsvzVMDPRvvrduDzX/oZqOu6qi17N7C5gfZXWyDK8tgDOFLD/8l6+4w1liP6GOCo1vq41roEWAeMrrbMaOBty+OPgSFKKWWZvk5rXay1PgEctWyvQerSWm/RWhdYniYAHerotX9RXVcxAvhaa52jtT4PfA2MtFFdE4G1dfTaV6W1/gHIucoio4F3tCEB8FJKtaV+99c169Jab7W8LjTc56s2++tKfslns67rasjP11mt9S7L40vAQaB9tcXq7TPWWIK+PXDK6nkal++kimW01iYgF/Ct5br1WZe1mRjf2OXclVKJSqkEpdT/1VFN11PXvZY/ET9WSnW8znXrsy4sTVxdgM1Wk+trf9XGlWqvz/11vap/vjSwSSm1Uyk1xwb19FNK7VVKfaWUCrVMs4v9pZRqgRGW660mN8j+UkazciSwrdqsevuMOczNwe2dUmoKEA0MtJrcWWt9WinVFdislNqvtT7WQCV9BqzVWhcrpR7C+GtocAO9dm1MAD7WWpdZTbPl/rJrSqlBGEF/q9XkWy37qw3wtVLqkOWItyHswvh95Sml7gQ+Bbo10GvXxt3Af7XW1kf/9b6/lFKtML5cntBaX6zLbV9NYzmiPw10tHrewTKtxmWUUi6AJ5Bdy3Xrsy6UUkOBp4F7tNbF5dO11qctP48D32F8yzdIXVrrbKtalgN9artufdZlZQLV/qyux/1VG1eqvT73V60opcIxfoejtdbZ5dOt9tc54BPqrsnymrTWF7XWeZbHXwKuSik/7GB/WVzt81Uv+0sp5YoR8u9rrf9dwyL19xmrjxMPdf0P4y+P4xh/ypefwAmttswjVD0Z+6HlcShVT8Yep+5OxtamrkiMk0/dqk33BppZHvsBKdTRSala1tXW6vEYIEFXnvg5YanP2/LYp6HqsizXE+PEmGqI/WX1GkFc+eTiXVQ9Uba9vvdXLevqhHHe6ZZq01sCHlaPtwIjG7CuwPLfH0ZgnrTsu1p9BuqrLst8T4x2/JYNtb8s7/0d4LWrLFNvn7E627n1/Q/jjPQRjNB82jJtIcZRMoA78JHlQ78d6Gq17tOW9Q4DdzRwXd8AGcAey78Nlum3APstH/T9wMwGrusvQJLl9bcAPa3WnWHZj0eB6Q1Zl+X5c8CiauvV9/5aC5wFSjHaQGcCDwMPW+YrYLGl7v1AdAPtr2vVtRw4b/X5SrRM72rZV3stv+enG7iuR60+XwlYfRHV9BloqLosy0zD6KBhvV59769bMc4B7LP6Xd3ZUJ8xGQJBCCEcXGNpoxdCCHGDJOiFEMLBSdALIYSDk6AXQggHJ0EvhBAOToJeCCEcnAS9EEI4uP8PjDk1O0q1r5wAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k6DkbNfXELbY"
      },
      "source": [
        "This is definitely starting to overfit by the third epoch, but the best dev set result is pretty impressive for such a small subset of the training data.\n",
        "\n",
        "(Caveat: by downsampling all of the data, we also ended up with a small dev set, so these results may have substantial variance.)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jwln4x2PELbZ"
      },
      "source": [
        "## Predict with the trained model\n",
        "\n",
        "Let's run a few simple cases through the trained model to illustrate its use."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "adJhCCYcELbZ"
      },
      "source": [
        "def predict_one(text):\n",
        "    # This function takes a string, vectorizes it with the tokenizer,\n",
        "    # and returns the model prediction. The vectorization follows\n",
        "    # the pattern we saw above for training data preparation:\n",
        "    tid, sid = tokenizer.encode(text, max_len=INPUT_LENGTH)\n",
        "    test_token_indices = [tid]\n",
        "    test_segment_ids = [sid]\n",
        "    test_X = [np.array(test_token_indices), np.array(test_segment_ids)]\n",
        "    # Run model.predict for this single case \n",
        "    predictions = model.predict(test_X)\n",
        "    # model.predict always returns a sequence. As we there's only a single input,\n",
        "    # we'll just return the value of the first (and only) element.\n",
        "    return predictions[0]\n",
        "\n",
        "\n",
        "# Get the index of the positive label (\"pos\"). Note that as LabelEncoder works\n",
        "# on lists, we need to wrap the label in a list when calling it, and take the\n",
        "# first (and only) element from the returned sequence.\n",
        "pos_index = label_encoder.transform(['pos'])[0]\n",
        "\n",
        "\n",
        "# Test the model with a few strings of our own.\n",
        "test_strings = [\n",
        "  'This movie is amazing!',\n",
        "  'This movie is amazingly bad!',\n",
        "  'The plot is boring, but the action is exciting.',\n",
        "  'The plot is exciting, but the action is boring.',\n",
        "  'I am not sure what to think about this movie.',\n",
        "]\n",
        "\n",
        "\n",
        "for s in test_strings:\n",
        "    pred = predict_one(s)\n",
        "    # The model returns the probability of both labels (pos/neg) where the\n",
        "    # neg = 1-pos (softmax). Just print out the positive label probability.\n",
        "    print('Positive probability {:.1%}\\t{}'.format(pred[pos_index], s))"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}