bert_sequence_labeling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sequence labeling with BERT\n",
    "\n",
    "This notebook briefly demonstrates fine-tuning a pretrained BERT model to a sequence labeling task.\n",
    "\n",
    "You probably want to run this notebook with GPU acceleration, as fine-tuning BERT on CPU can be fairly slow even with a comparatively small dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration\n",
    "\n",
    "The following variables configure a few aspects of the data, model, and training process. To adapt this example to a different dataset, you'll probably want to change these to match.\n",
    "\n",
    "Note in particular that we're limiting the number of examples and the maximum sequence length to make training faster. \n",
    "\n",
    "When running on GPU, it's also necessary to make sure that the input length and batch size are not so large as to cause a batch to exceed GPU memory. If you're getting a message like `Resource exhausted: OOM when allocating tensor`, try smaller numbers for these."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Maximum number of examples to read\n",
    "MAX_EXAMPLES = 2000\n",
    "\n",
    "# Maximum length of input sequence in tokens\n",
    "INPUT_LENGTH = 25\n",
    "\n",
    "# Number of epochs to train for\n",
    "EPOCHS = 3\n",
    "\n",
    "# Optimizer learning rate\n",
    "LEARNING_RATE = 0.00002\n",
    "\n",
    "# Training batch size\n",
    "BATCH_SIZE = 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Package setup\n",
    "\n",
    "We'll use [keras-bert](https://github.com/CyberZHG/keras-bert). Make sure the package is installed. (`pip` is the Python [package installer](https://pip.pypa.io/en/stable/))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: keras-bert in /Users/smp/Library/Python/3.7/lib/python/site-packages (0.81.0)\n",
      "Requirement already satisfied: Keras in /Users/smp/Library/Python/3.7/lib/python/site-packages (from keras-bert) (2.3.1)\n",
      "Requirement already satisfied: keras-transformer>=0.30.0 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from keras-bert) (0.32.0)\n",
      "Requirement already satisfied: numpy in /Users/smp/Library/Python/3.7/lib/python/site-packages (from keras-bert) (1.18.1)\n",
      "Requirement already satisfied: scipy>=0.14 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from Keras->keras-bert) (1.4.1)\n",
      "Requirement already satisfied: keras-preprocessing>=1.0.5 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from Keras->keras-bert) (1.1.0)\n",
      "Requirement already satisfied: h5py in /Users/smp/Library/Python/3.7/lib/python/site-packages (from Keras->keras-bert) (2.10.0)\n",
      "Requirement already satisfied: keras-applications>=1.0.6 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from Keras->keras-bert) (1.0.8)\n",
      "Requirement already satisfied: six>=1.9.0 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from Keras->keras-bert) (1.14.0)\n",
      "Requirement already satisfied: pyyaml in /Users/smp/Library/Python/3.7/lib/python/site-packages (from Keras->keras-bert) (5.3)\n",
      "Requirement already satisfied: keras-pos-embd>=0.10.0 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from keras-transformer>=0.30.0->keras-bert) (0.11.0)\n",
      "Requirement already satisfied: keras-position-wise-feed-forward>=0.5.0 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from keras-transformer>=0.30.0->keras-bert) (0.6.0)\n",
      "Requirement already satisfied: keras-embed-sim>=0.7.0 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from keras-transformer>=0.30.0->keras-bert) (0.7.0)\n",
      "Requirement already satisfied: keras-layer-normalization>=0.12.0 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from keras-transformer>=0.30.0->keras-bert) (0.14.0)\n",
      "Requirement already satisfied: keras-multi-head>=0.22.0 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from keras-transformer>=0.30.0->keras-bert) (0.22.0)\n",
      "Requirement already satisfied: keras-self-attention==0.41.0 in /Users/smp/Library/Python/3.7/lib/python/site-packages (from keras-multi-head>=0.22.0->keras-transformer>=0.30.0->keras-bert) (0.41.0)\n"
     ]
    }
   ],
   "source": [
    "!pip3 install keras-bert"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensorflow configuration\n",
    "\n",
    "We'll need set an environment variable for keras-bert to use `tensorflow.python.keras`. (This is a technical detail that is not related to the model.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "\n",
    "os.environ['TF_KERAS'] = '1'    # Required to use tensorflow.python.keras with keras-bert"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download dataset\n",
    "\n",
    "We'll use part-of-speech tags from a recent [Universal Dependencies]() version of the English [EWT corpus](https://nlp.stanford.edu/pubs/Gold_LREC14.pdf).\n",
    "\n",
    "Download the training and development subsets of the corpus data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File ‘en_ewt-ud-train.conllu’ already there; not retrieving.\n",
      "\n",
      "File ‘en_ewt-ud-dev.conllu’ already there; not retrieving.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Give -nc (--no-clobber) argument so that the file isn't downloaded multiple times \n",
    "!wget -nc https://github.com/UniversalDependencies/UD_English-EWT/raw/master/en_ewt-ud-train.conllu\n",
    "!wget -nc https://github.com/UniversalDependencies/UD_English-EWT/raw/master/en_ewt-ud-dev.conllu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data\n",
    "\n",
    "The corpus is in [CoNLL-U](https://universaldependencies.org/format) format. We won't get into the details of the format here, as it's enough to know the following:\n",
    "\n",
    "* Lines beginning with `#` are comments and can be ignored\n",
    "* Empty lines delimit sentences\n",
    "* Remaining lines contain information on words, with fields separated by a tab\n",
    "* On lines where the first field is an integer (word index), the second field contains the word surface form, and the fourth the part-of-speech tag\n",
    "\n",
    "We'll grab the wordforms and POS tags and keep them organized by sentences.\n",
    "\n",
    "Note that before (potentially) truncating the examples to `MAX_EXAMPLES` we'll shuffle the list of sentences, but keep the words in their original order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: truncating examples in en_ewt-ud-train.conllu from 12543 to 2000\n",
      "Note: truncating examples in en_ewt-ud-dev.conllu from 2002 to 2000\n",
      "\n",
      "Example sentence:\n",
      "They\tPRON\n",
      "had\tVERB\n",
      "a\tDET\n",
      "great\tADJ\n",
      "selection\tNOUN\n",
      "of\tADP\n",
      "colors\tNOUN\n",
      "to\tPART\n",
      "choose\tVERB\n",
      "from\tADP\n",
      "and\tCCONJ\n",
      "their\tPRON\n",
      "seats\tNOUN\n",
      "are\tAUX\n",
      "super\tADV\n",
      "comfty\tADJ\n",
      ".\tPUNCT\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "\n",
    "random.seed(1234)    # This makes random.shuffle() produce the same order every time\n",
    "\n",
    "\n",
    "\n",
    "def load_conllu_tags(fn):\n",
    "    sentences = []\n",
    "    with open(fn) as f:\n",
    "        current = []\n",
    "        for line in f:\n",
    "            line = line.rstrip('\\n')\n",
    "            if line.startswith('#'):\n",
    "                continue    # Comment line, skip\n",
    "            if line == '' or line.isspace():\n",
    "                # Empty line, sentence boundary\n",
    "                sentences.append(current)\n",
    "                current = []\n",
    "                continue\n",
    "            fields = line.split('\\t')\n",
    "            if not fields[0].isdigit():\n",
    "                continue    # Not a regular word, skip\n",
    "            word, tag = fields[1], fields[3]\n",
    "            current.append((word, tag))\n",
    "            \n",
    "    random.shuffle(sentences)\n",
    "    if len(sentences) > MAX_EXAMPLES:\n",
    "        print('Note: truncating examples in {} from {} to {}'.format(fn, len(sentences), MAX_EXAMPLES))\n",
    "        sentences = sentences[:MAX_EXAMPLES]\n",
    "\n",
    "    return sentences\n",
    "\n",
    "\n",
    "train_sentences = load_conllu_tags('en_ewt-ud-train.conllu')\n",
    "dev_sentences = load_conllu_tags('en_ewt-ud-dev.conllu')\n",
    "\n",
    "\n",
    "# Have a look at the data\n",
    "print('\\nExample sentence:')\n",
    "for word, tag in train_sentences[0]:\n",
    "    print('{}\\t{}'.format(word, tag))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download pretrained BERT model\n",
    "\n",
    "As training BERT from scratch generally takes days, we'll here load a pretrained model and fine-tune it for our task. URLs to download pre-trained models made available by Google are found at https://github.com/google-research/bert .\n",
    "\n",
    "`cased_L-12_H-768_A-12` is a case-sensitive BERT \"base\" model for English: 12 layers, 768-dimensional hidden state, and 12 \"heads\" for multi-head attention."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File ‘cased_L-12_H-768_A-12.zip’ already there; not retrieving.\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "# Give -nc (--no-clobber) argument so that the file isn't downloaded multiple times \n",
    "!wget -nc https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unpack the downloaded file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  cased_L-12_H-768_A-12.zip\r\n"
     ]
    }
   ],
   "source": [
    "# Give -n argument so that existing files aren't overwritten \n",
    "!unzip -n cased_L-12_H-768_A-12.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are three key parts to the package contents:\n",
    "\n",
    "* `vocab.txt`: plain text file listing vocabulary items\n",
    "* `bert_config.json`: model configuration in JSON format\n",
    "* `bert_model.ckpt.*`: model checkpoint data with pre-trained weights in [Tensorflow checkpoint format](https://www.tensorflow.org/guide/checkpoint)\n",
    "\n",
    "Take note of the path to these (you'll need to change this if you pick a different model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "bert_vocab_path = 'cased_L-12_H-768_A-12/vocab.txt'\n",
    "bert_config_path = 'cased_L-12_H-768_A-12/bert_config.json'\n",
    "bert_checkpoint_path = 'cased_L-12_H-768_A-12/bert_model.ckpt'    # suffixes not required"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also take note if the model we downloaded was a case-sensitive (cased) or not. (This must match the model.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_is_cased = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load BERT vocabulary\n",
    "\n",
    "This is just a plain text file with one vocabulary item per line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[PAD]', 'щ', '吉', 'told', 'space', 'operations', 'proposed', 'Oxford', 'showing', 'domestic', 'mountains', 'commission', 'voices', 'associate', 'hills', 'Guide', 'relaxed', 'Page', 'Heights', 'singers', 'Interior', 'considers', 'facilitate', 'shouting', '1826', 'constitute', 'alter', 'clip', 'Into', 'Memory', 'ballad', 'Owens', 'Langdon', 'aquatic', 'stereo', 'Cass', 'Shock', '195', '##tec', '##sonic', 'attested', '##rdes', '1840s', '##90', 'Guys', '##rien', 'Munro', 'Ursula', 'mesh', 'diplomacy', 'Newmarket', '##oughs', 'synthesizers', 'Drugs', 'monstrous', '##ynamic', 'troll', '##ٹ']\n"
     ]
    }
   ],
   "source": [
    "vocab = []\n",
    "with open(bert_vocab_path) as f:\n",
    "    for i, line in enumerate(f):\n",
    "        vocab.append(line.rstrip('\\n'))    # rstrip to remove newline characters\n",
    "\n",
    "\n",
    "# Print a list with every 500th vocabulary item\n",
    "print(vocab[0::500])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load BERT configuration\n",
    "\n",
    "The configuration is just a JSON file, so we can read it in with `json.load` from the python `json` library.\n",
    "\n",
    "We won't actually need to use these configuration details directly (keras-bert takes care of them for us), so this is just here to show what information is contained in the config file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'attention_probs_dropout_prob': 0.1,\n",
      " 'hidden_act': 'gelu',\n",
      " 'hidden_dropout_prob': 0.1,\n",
      " 'hidden_size': 768,\n",
      " 'initializer_range': 0.02,\n",
      " 'intermediate_size': 3072,\n",
      " 'max_position_embeddings': 512,\n",
      " 'num_attention_heads': 12,\n",
      " 'num_hidden_layers': 12,\n",
      " 'type_vocab_size': 2,\n",
      " 'vocab_size': 28996}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "from pprint import pprint    # pretty-printer for output\n",
    "\n",
    "\n",
    "with open(bert_config_path) as f:\n",
    "    config = json.load(f)\n",
    "\n",
    "\n",
    "# Print configuration contents\n",
    "pprint(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create BERT tokenizer\n",
    "\n",
    "To create the tokenizer, we'll need a mapping from vocabulary items to their corresponding integer indices. We do this conventionally using `enumerate`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'##umi': 14088,\n",
      " 'Helsinki': 12471,\n",
      " 'assassinated': 17493,\n",
      " 'brackets': 22019,\n",
      " 'drowned': 14100,\n",
      " 'educational': 4339,\n",
      " 'firmly': 7487,\n",
      " 'instincts': 17477,\n",
      " 'internet': 7210,\n",
      " 'nuts': 13937}\n"
     ]
    }
   ],
   "source": [
    "# Create mapping from vocabulary items to their indices in the vocabulary\n",
    "token_dict = { v: i for i, v in enumerate(vocab) }\n",
    "\n",
    "\n",
    "# Print some random examples of the mapping\n",
    "pprint(dict(random.choices(list(token_dict.items()), k=10)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use the keras-bert `Tokenizer` for BERT tokenization. The implementation supports\n",
    "\n",
    "* (Optional) lowercasing: `Hello` → `hello`\n",
    "* Basic tokenization: `Hello!` → `Hello` `!`, `multi-part` → `multi` `-` `part`\n",
    "* Wordpiece tokenization: `comprehensively` → `comprehensive` `##ly`\n",
    "* Adding special tokens: `Sentence`  → `[CLS]` `Sentence` `[SEP]`\n",
    "* Mapping to integer indices\n",
    "* Generating segment sequence\n",
    "* (Optional) padding and truncation to length\n",
    "\n",
    "In the following example, notice how words not in the dictionary are broken up into subwords (with continuation parts starting with `##`) and how unknown _characters_ are mapped to a special unknown word token `[UNK]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original string: Hello BERT!\n",
      "Tokenized: ['[CLS]', 'Hello', 'B', '##ER', '##T', '!', '[SEP]']\n",
      "Encoded: [101, 8667, 139, 9637, 1942, 106, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      "Segments: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      "Decoded: Hello B ##ER ##T !\n",
      "\n",
      "Original string: Unknown: 你\n",
      "Tokenized: ['[CLS]', 'Unknown', ':', '你', '[SEP]']\n",
      "Encoded: [101, 16285, 131, 100, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      "Segments: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      "Decoded: Unknown : [UNK]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from keras_bert import Tokenizer\n",
    "\n",
    "\n",
    "tokenizer = Tokenizer(token_dict, cased=model_is_cased)\n",
    "\n",
    "\n",
    "# Let's test that out\n",
    "for s in ['Hello BERT!', 'Unknown: 你']:\n",
    "    print('Original string:', s)\n",
    "    print('Tokenized:', tokenizer.tokenize(s))\n",
    "    indices, segments = tokenizer.encode(s, max_len=20)    # max_len for padding and truncation\n",
    "    print('Encoded:', indices)\n",
    "    print('Segments:', segments)\n",
    "    print('Decoded:', ' '.join(tokenizer.decode(indices)))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vectorize data\n",
    "\n",
    "We'll use keras-bert `Tokenizer` to tokenize and vectorize words. For vectorizing labels (tags), we'll create a simple mapping ourselves.\n",
    "\n",
    "We'll also add a special placeholder label `NO_LABEL` for padding and continuation wordpieces (more on that below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of unique labels: 18\n",
      "Tags: ['NONE', 'ADJ', 'ADP', 'ADV', 'AUX', 'CCONJ', 'DET', 'INTJ', 'NOUN', 'NUM', 'PART', 'PRON', 'PROPN', 'PUNCT', 'SCONJ', 'SYM', 'VERB', 'X']\n",
      "Mapping: {'NONE': 0, 'ADJ': 1, 'ADP': 2, 'ADV': 3, 'AUX': 4, 'CCONJ': 5, 'DET': 6, 'INTJ': 7, 'NOUN': 8, 'NUM': 9, 'PART': 10, 'PRON': 11, 'PROPN': 12, 'PUNCT': 13, 'SCONJ': 14, 'SYM': 15, 'VERB': 16, 'X': 17}\n",
      "Inverted: {0: 'NONE', 1: 'ADJ', 2: 'ADP', 3: 'ADV', 4: 'AUX', 5: 'CCONJ', 6: 'DET', 7: 'INTJ', 8: 'NOUN', 9: 'NUM', 10: 'PART', 11: 'PRON', 12: 'PROPN', 13: 'PUNCT', 14: 'SCONJ', 15: 'SYM', 16: 'VERB', 17: 'X'}\n"
     ]
    }
   ],
   "source": [
    "# This is our special placeholder label value\n",
    "NO_LABEL = 'NONE'\n",
    "\n",
    "\n",
    "# This just flattens out tags from the sentences list-of-lists\n",
    "train_tags = [tag for sentence in train_sentences for word, tag in sentence]\n",
    "# Use set() to get the unique tags and add our special label to the list\n",
    "unique_tags = [NO_LABEL] + sorted(set(train_tags))\n",
    "\n",
    "# Create mappings from tags to integer values and back\n",
    "tag_to_int = { t: i for i, t in enumerate(unique_tags) }\n",
    "int_to_tag = { i: t for t, i in tag_to_int.items() }\n",
    "\n",
    "# Take note of how many unique labels (tags) there are in the data\n",
    "num_labels = len(unique_tags)\n",
    "\n",
    "\n",
    "# Let's see what we got\n",
    "print('Number of unique labels:', num_labels)\n",
    "print('Tags:', unique_tags)\n",
    "print('Mapping:', tag_to_int)\n",
    "print('Inverted:', int_to_tag)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The source data has one tag for each (syntactic) word. If we were training a model from scratch, we could define the tokenization to match the data so that we would get a simply one-to-one mapping between input words and output tags. However, as we're using a pre-trained model, we need to work with the existing tokenization, which may split up some of the input words into multiple parts.\n",
    "\n",
    "For example, if the data contains the word, tag pair `(complicatedly, ADV)` and the tokenizer splits up the word into `complicated` `##ly`, we have two tokens (`complicated` and `##ly`) but just one tag (`ADV`). There are a number of ways we could resolve this. Here, we will assign the original tag to the first wordpiece and the placeholder tag we introduced earlier to any subsequent pieces, so e.g. `(complicatedly, ADV)` would map to `(complicated, ADV)`, `(##ly, NONE)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['They'] ['PRON']\n",
      "['had'] ['VERB']\n",
      "['a'] ['DET']\n",
      "['great'] ['ADJ']\n",
      "['selection'] ['NOUN']\n",
      "['of'] ['ADP']\n",
      "['colors'] ['NOUN']\n",
      "['to'] ['PART']\n",
      "['choose'] ['VERB']\n",
      "['from'] ['ADP']\n",
      "['and'] ['CCONJ']\n",
      "['their'] ['PRON']\n",
      "['seats'] ['NOUN']\n",
      "['are'] ['AUX']\n",
      "['super'] ['ADV']\n",
      "['com', '##fty'] ['ADJ', 'NONE']\n",
      "['.'] ['PUNCT']\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "\n",
    "def tokenize_word(word):\n",
    "    # keras-bert tokenizer wraps its inputs with the special BERT tokens\n",
    "    # [CLS] and [SEP]. To tokenize individual words, we remove these.\n",
    "    wrapped = tokenizer.tokenize(word)\n",
    "    return wrapped[1:-1]    # remove [CLS] at the first and [SEP] at the last position\n",
    "\n",
    "\n",
    "def tokenize_sentence(sentence):\n",
    "    tokenized = []\n",
    "    for word, tag in sentence:\n",
    "        tokens = tokenize_word(word)\n",
    "        # Original tag for first wordpiece, fill in placeholder tags for\n",
    "        # any continuation pieces.\n",
    "        tags = [tag] + [NO_LABEL] * (len(tokens)-1)\n",
    "        tokenized.append((tokens, tags))\n",
    "    return tokenized\n",
    "\n",
    "\n",
    "def tokenize_sentences(sentences):\n",
    "    return [tokenize_sentence(s) for s in sentences]\n",
    "\n",
    "\n",
    "tokenized_train = tokenize_sentences(train_sentences)\n",
    "tokenized_dev = tokenize_sentences(dev_sentences)\n",
    "\n",
    "\n",
    "# Let's look at an example sentence after tokenization\n",
    "for tokens, tags in tokenized_train[0]:\n",
    "    print(tokens, tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To vectorize a tokenized sentence and its labels, we'll truncate to target length, add the special BERT tokens `[CLS]` and `[SEP]` (and corresponding placeholder labels), pad to length with the special `[PAD]` token if necessary, and map both tokens and tags to integer indices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_X[0].shape: (2000, 25)\n",
      "train_X[1].shape: (2000, 25)\n",
      "train_Y[0].shape: (25, 18)\n",
      "train_X[0][0] (first sentence token IDs):\n",
      "[  101  1220  1125   170  1632  4557  1104  5769  1106  4835  1121  1105\n",
      "  1147  3474  1132  7688  3254 27944   119   102     0     0     0     0\n",
      "     0]\n",
      "train_X[1][0] (first sentence segment IDs):\n",
      "[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
      "train_Y[0][0] (first sentence tag IDs):\n",
      "[[1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0.]\n",
      " [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]\n",
      " [1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]\n",
      "decoded train_X[0][0] (first sentence tokens):\n",
      "[CLS] They had a great selection of colors to choose from and their seats are super com ##fty . [SEP] [PAD] [PAD] [PAD] [PAD] [PAD]\n",
      "decoded train_Y[0] (first sentence tags):\n",
      "NONE PRON VERB DET ADJ NOUN ADP NOUN PART VERB ADP CCONJ PRON NOUN AUX ADV ADJ NONE PUNCT NONE NONE NONE NONE NONE NONE\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.utils import to_categorical\n",
    "\n",
    "\n",
    "def vectorize_tokenized(tokenized, length=INPUT_LENGTH):\n",
    "    # Flatten lists-of-lists of tokens and tags\n",
    "    token_sequence, tag_sequence = [], []\n",
    "    for tokens, tags in tokenized:\n",
    "        token_sequence.extend(tokens)\n",
    "        tag_sequence.extend(tags)\n",
    "\n",
    "    # Truncation and padding using length-2 to leave space for\n",
    "    # wrapping with the special [CLS] and [SEP] tokens\n",
    "    if len(token_sequence) > length-2:\n",
    "        token_sequence = token_sequence[:length-2]\n",
    "        tag_sequence = tag_sequence[:length-2]\n",
    "\n",
    "    # Wrap with [CLS] and [SEP], adding corresponding placeholder labels\n",
    "    token_sequence = ['[CLS]'] + token_sequence + ['[SEP]']\n",
    "    tag_sequence = [NO_LABEL] + tag_sequence + [NO_LABEL]\n",
    "\n",
    "    while len(token_sequence) < length:\n",
    "        token_sequence.append('[PAD]')\n",
    "        tag_sequence.append(NO_LABEL)\n",
    "    \n",
    "    # Use keras-bert tokenizer and previously created label mapping\n",
    "    token_ids = tokenizer._convert_tokens_to_ids(token_sequence)\n",
    "    tag_ids = [tag_to_int[tag] for tag in tag_sequence]\n",
    "    \n",
    "    # Also create all-zeros segment IDs for BERT\n",
    "    segment_ids = [0] * INPUT_LENGTH\n",
    "\n",
    "    return token_ids, segment_ids, tag_ids\n",
    "\n",
    "\n",
    "def vectorize_dataset(tokenized_data, length=INPUT_LENGTH):\n",
    "    # Create separate lists of token ID, segment ID, and tag ID lists,\n",
    "    # each of the inner lists representing one sentence.\n",
    "    token_ids_list, segment_ids_list, tag_ids_list = [], [], []\n",
    "    for tokenized in tokenized_data:\n",
    "        token_ids, segment_ids, tag_ids = vectorize_tokenized(tokenized, length=length)\n",
    "        token_ids_list.append(token_ids)\n",
    "        segment_ids_list.append(segment_ids)\n",
    "        tag_ids_list.append(tag_ids)\n",
    "\n",
    "    # Return as numpy arrays. Input (X) consists of the token and\n",
    "    # segment IDs, output (Y) of the tag ids. We'll use a one-hot\n",
    "    # representation for the output.\n",
    "    X = [np.array(token_ids_list), np.array(segment_ids_list)]\n",
    "    Y = to_categorical(np.array(tag_ids_list))\n",
    "\n",
    "    return X, Y\n",
    "    \n",
    "\n",
    "train_X, train_Y = vectorize_dataset(tokenized_train)\n",
    "dev_X, dev_Y = vectorize_dataset(tokenized_dev)\n",
    "\n",
    "\n",
    "# Let's have a bit of a look at that\n",
    "print('train_X[0].shape:', train_X[0].shape)\n",
    "print('train_X[1].shape:', train_X[1].shape)\n",
    "print('train_Y[0].shape:', train_Y[0].shape)\n",
    "print('train_X[0][0] (first sentence token IDs):')\n",
    "print(train_X[0][0])\n",
    "print('train_X[1][0] (first sentence segment IDs):')\n",
    "print(train_X[1][0])\n",
    "print('train_Y[0][0] (first sentence tag IDs):')\n",
    "print(train_Y[0])\n",
    "print('decoded train_X[0][0] (first sentence tokens):')\n",
    "print(' '.join([tokenizer._token_dict_inv[t] for t in train_X[0][0]]))\n",
    "print('decoded train_Y[0] (first sentence tags):')\n",
    "print(' '.join([int_to_tag[t.argmax()] for t in train_Y[0]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load pretrained BERT model\n",
    "\n",
    "We'll use the keras-bert function `load_trained_model_from_checkpoint` to load the model from the checkpoint we downloaded earlier.\n",
    "\n",
    "Explanation for a few parameters from keras-bert documentation:\n",
    "\n",
    "* `training`: If `training`, the whole model will be returned. Otherwise, the MLM [masked language modeling] and NSP [next sentence prediction] parts will be ignored.\n",
    "* `trainable`: Whether the model is trainable. The default value is the same with `training`.\n",
    "\n",
    "We don't need the masked language modeling or next sentence prediction parts (these are primarily for pretraining), so we'll use `training=False` but `trainable=True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/smp/Library/Python/3.7/lib/python/site-packages/tensorflow_core/python/keras/initializers.py:119: calling RandomUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
      "WARNING:tensorflow:From /Users/smp/Library/Python/3.7/lib/python/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n"
     ]
    }
   ],
   "source": [
    "from keras_bert import load_trained_model_from_checkpoint\n",
    "\n",
    "\n",
    "pretrained_model = load_trained_model_from_checkpoint(\n",
    "    config_file = bert_config_path,\n",
    "    checkpoint_file = bert_checkpoint_path,\n",
    "    training = False,\n",
    "    trainable = True,\n",
    "    seq_len = INPUT_LENGTH\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a bit of a look at that model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor 'Input-Token:0' shape=(?, 25) dtype=float32>,\n",
       " <tf.Tensor 'Input-Segment:0' shape=(?, 25) dtype=float32>]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This is a keras model, so we can figure out what inputs it takes like so:\n",
    "pretrained_model.inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor 'Encoder-12-FeedForward-Norm/add_1:0' shape=(?, 25, 768) dtype=float32>]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# And similarly for outputs:\n",
    "pretrained_model.outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, the model takes `Input-Token` and `Input-Segment` inputs, both of dimension (batch-size, input-length), and produces a single output tensor of dimension (batch-size, input-length, hidden-dim). This is just what we want for sequence labeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_1\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "Input-Token (InputLayer)        [(None, 25)]         0                                            \n",
      "__________________________________________________________________________________________________\n",
      "Input-Segment (InputLayer)      [(None, 25)]         0                                            \n",
      "__________________________________________________________________________________________________\n",
      "Embedding-Token (TokenEmbedding [(None, 25, 768), (2 22268928    Input-Token[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "Embedding-Segment (Embedding)   (None, 25, 768)      1536        Input-Segment[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "Embedding-Token-Segment (Add)   (None, 25, 768)      0           Embedding-Token[0][0]            \n",
      "                                                                 Embedding-Segment[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "Embedding-Position (PositionEmb (None, 25, 768)      19200       Embedding-Token-Segment[0][0]    \n",
      "__________________________________________________________________________________________________\n",
      "Embedding-Dropout (Dropout)     (None, 25, 768)      0           Embedding-Position[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "Embedding-Norm (LayerNormalizat (None, 25, 768)      1536        Embedding-Dropout[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-MultiHeadSelfAttentio (None, 25, 768)      2362368     Embedding-Norm[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-MultiHeadSelfAttentio (None, 25, 768)      0           Encoder-1-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-MultiHeadSelfAttentio (None, 25, 768)      0           Embedding-Norm[0][0]             \n",
      "                                                                 Encoder-1-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-MultiHeadSelfAttentio (None, 25, 768)      1536        Encoder-1-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-FeedForward (FeedForw (None, 25, 768)      4722432     Encoder-1-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-FeedForward-Dropout ( (None, 25, 768)      0           Encoder-1-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-FeedForward-Add (Add) (None, 25, 768)      0           Encoder-1-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-1-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-FeedForward-Norm (Lay (None, 25, 768)      1536        Encoder-1-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-MultiHeadSelfAttentio (None, 25, 768)      2362368     Encoder-1-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-MultiHeadSelfAttentio (None, 25, 768)      0           Encoder-2-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-MultiHeadSelfAttentio (None, 25, 768)      0           Encoder-1-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-2-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-MultiHeadSelfAttentio (None, 25, 768)      1536        Encoder-2-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-FeedForward (FeedForw (None, 25, 768)      4722432     Encoder-2-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-FeedForward-Dropout ( (None, 25, 768)      0           Encoder-2-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-FeedForward-Add (Add) (None, 25, 768)      0           Encoder-2-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-2-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-FeedForward-Norm (Lay (None, 25, 768)      1536        Encoder-2-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-MultiHeadSelfAttentio (None, 25, 768)      2362368     Encoder-2-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-MultiHeadSelfAttentio (None, 25, 768)      0           Encoder-3-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-MultiHeadSelfAttentio (None, 25, 768)      0           Encoder-2-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-3-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-MultiHeadSelfAttentio (None, 25, 768)      1536        Encoder-3-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-FeedForward (FeedForw (None, 25, 768)      4722432     Encoder-3-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-FeedForward-Dropout ( (None, 25, 768)      0           Encoder-3-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-FeedForward-Add (Add) (None, 25, 768)      0           Encoder-3-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-3-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-FeedForward-Norm (Lay (None, 25, 768)      1536        Encoder-3-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-MultiHeadSelfAttentio (None, 25, 768)      2362368     Encoder-3-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-MultiHeadSelfAttentio (None, 25, 768)      0           Encoder-4-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-MultiHeadSelfAttentio (None, 25, 768)      0           Encoder-3-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-4-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-MultiHeadSelfAttentio (None, 25, 768)      1536        Encoder-4-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-FeedForward (FeedForw (None, 25, 768)      4722432     Encoder-4-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-FeedForward-Dropout ( (None, 25, 768)      0           Encoder-4-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-FeedForward-Add (Add) (None, 25, 768)      0           Encoder-4-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-4-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-FeedForward-Norm (Lay (None, 25, 768)      1536        Encoder-4-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-MultiHeadSelfAttentio (None, 25, 768)      2362368     Encoder-4-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-MultiHeadSelfAttentio (None, 25, 768)      0           Encoder-5-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-MultiHeadSelfAttentio (None, 25, 768)      0           Encoder-4-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-5-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-MultiHeadSelfAttentio (None, 25, 768)      1536        Encoder-5-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-FeedForward (FeedForw (None, 25, 768)      4722432     Encoder-5-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-FeedForward-Dropout ( (None, 25, 768)      0           Encoder-5-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-FeedForward-Add (Add) (None, 25, 768)      0           Encoder-5-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-5-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-FeedForward-Norm (Lay (None, 25, 768)      1536        Encoder-5-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-MultiHeadSelfAttentio (None, 25, 768)      2362368     Encoder-5-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-MultiHeadSelfAttentio (None, 25, 768)      0           Encoder-6-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-MultiHeadSelfAttentio (None, 25, 768)      0           Encoder-5-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-6-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-MultiHeadSelfAttentio (None, 25, 768)      1536        Encoder-6-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-FeedForward (FeedForw (None, 25, 768)      4722432     Encoder-6-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-FeedForward-Dropout ( (None, 25, 768)      0           Encoder-6-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-FeedForward-Add (Add) (None, 25, 768)      0           Encoder-6-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-6-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-FeedForward-Norm (Lay (None, 25, 768)      1536        Encoder-6-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-MultiHeadSelfAttentio (None, 25, 768)      2362368     Encoder-6-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-MultiHeadSelfAttentio (None, 25, 768)      0           Encoder-7-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-MultiHeadSelfAttentio (None, 25, 768)      0           Encoder-6-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-7-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-MultiHeadSelfAttentio (None, 25, 768)      1536        Encoder-7-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-FeedForward (FeedForw (None, 25, 768)      4722432     Encoder-7-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-FeedForward-Dropout ( (None, 25, 768)      0           Encoder-7-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-FeedForward-Add (Add) (None, 25, 768)      0           Encoder-7-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-7-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-FeedForward-Norm (Lay (None, 25, 768)      1536        Encoder-7-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-MultiHeadSelfAttentio (None, 25, 768)      2362368     Encoder-7-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-MultiHeadSelfAttentio (None, 25, 768)      0           Encoder-8-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-MultiHeadSelfAttentio (None, 25, 768)      0           Encoder-7-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-8-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-MultiHeadSelfAttentio (None, 25, 768)      1536        Encoder-8-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-FeedForward (FeedForw (None, 25, 768)      4722432     Encoder-8-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-FeedForward-Dropout ( (None, 25, 768)      0           Encoder-8-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-FeedForward-Add (Add) (None, 25, 768)      0           Encoder-8-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-8-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-FeedForward-Norm (Lay (None, 25, 768)      1536        Encoder-8-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-MultiHeadSelfAttentio (None, 25, 768)      2362368     Encoder-8-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-MultiHeadSelfAttentio (None, 25, 768)      0           Encoder-9-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-MultiHeadSelfAttentio (None, 25, 768)      0           Encoder-8-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-9-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-MultiHeadSelfAttentio (None, 25, 768)      1536        Encoder-9-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-FeedForward (FeedForw (None, 25, 768)      4722432     Encoder-9-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-FeedForward-Dropout ( (None, 25, 768)      0           Encoder-9-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-FeedForward-Add (Add) (None, 25, 768)      0           Encoder-9-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-9-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-FeedForward-Norm (Lay (None, 25, 768)      1536        Encoder-9-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-MultiHeadSelfAttenti (None, 25, 768)      2362368     Encoder-9-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-MultiHeadSelfAttenti (None, 25, 768)      0           Encoder-10-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-MultiHeadSelfAttenti (None, 25, 768)      0           Encoder-9-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-10-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-MultiHeadSelfAttenti (None, 25, 768)      1536        Encoder-10-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-FeedForward (FeedFor (None, 25, 768)      4722432     Encoder-10-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-FeedForward-Dropout  (None, 25, 768)      0           Encoder-10-FeedForward[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-FeedForward-Add (Add (None, 25, 768)      0           Encoder-10-MultiHeadSelfAttention\n",
      "                                                                 Encoder-10-FeedForward-Dropout[0]\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-FeedForward-Norm (La (None, 25, 768)      1536        Encoder-10-FeedForward-Add[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-MultiHeadSelfAttenti (None, 25, 768)      2362368     Encoder-10-FeedForward-Norm[0][0]\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-MultiHeadSelfAttenti (None, 25, 768)      0           Encoder-11-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-MultiHeadSelfAttenti (None, 25, 768)      0           Encoder-10-FeedForward-Norm[0][0]\n",
      "                                                                 Encoder-11-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-MultiHeadSelfAttenti (None, 25, 768)      1536        Encoder-11-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-FeedForward (FeedFor (None, 25, 768)      4722432     Encoder-11-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-FeedForward-Dropout  (None, 25, 768)      0           Encoder-11-FeedForward[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-FeedForward-Add (Add (None, 25, 768)      0           Encoder-11-MultiHeadSelfAttention\n",
      "                                                                 Encoder-11-FeedForward-Dropout[0]\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-FeedForward-Norm (La (None, 25, 768)      1536        Encoder-11-FeedForward-Add[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-MultiHeadSelfAttenti (None, 25, 768)      2362368     Encoder-11-FeedForward-Norm[0][0]\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-MultiHeadSelfAttenti (None, 25, 768)      0           Encoder-12-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-MultiHeadSelfAttenti (None, 25, 768)      0           Encoder-11-FeedForward-Norm[0][0]\n",
      "                                                                 Encoder-12-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-MultiHeadSelfAttenti (None, 25, 768)      1536        Encoder-12-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-FeedForward (FeedFor (None, 25, 768)      4722432     Encoder-12-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-FeedForward-Dropout  (None, 25, 768)      0           Encoder-12-FeedForward[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-FeedForward-Add (Add (None, 25, 768)      0           Encoder-12-MultiHeadSelfAttention\n",
      "                                                                 Encoder-12-FeedForward-Dropout[0]\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-FeedForward-Norm (La (None, 25, 768)      1536        Encoder-12-FeedForward-Add[0][0] \n",
      "==================================================================================================\n",
      "Total params: 107,345,664\n",
      "Trainable params: 107,345,664\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "pretrained_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a regular Keras model. In Keras, models behave very much like layers, so we're able to wrap this in our own model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build sequence labeling model\n",
    "\n",
    "We'll make a _very_ simple model for sequence labeling: just attach a time-distributed dense layer with softmax activation to the output, and connect the model inputs to the BERT model inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"Encoder-12-FeedForward-Norm/add_1:0\", shape=(?, 25, 768), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# model.outputs is a list, here with a single item. We'll\n",
    "# add our output layer on top of that.\n",
    "bert_out = pretrained_model.outputs[0]\n",
    "\n",
    "print(bert_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model is standard Keras, where the pretrained BERT model is behaving essentially as a layer of our \"wrapping\" model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.layers import TimeDistributed, Dense\n",
    "\n",
    "\n",
    "out = TimeDistributed(Dense(num_labels, activation='softmax'))(bert_out)\n",
    "model = Model(\n",
    "    inputs=pretrained_model.inputs,\n",
    "    outputs=[out]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create optimizer\n",
    "\n",
    "BERT is pretrained with an Adam optimizer with warmup and regularization using weight decay. We won't go into detail on these optimizer settings, but will instead largely copy parameters used in the original BERT work.\n",
    "\n",
    "(If you're interested in tuning the training process, trying different values of `LEARNING_RATE` is a good place to start!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras_bert import calc_train_steps, AdamWarmup\n",
    "\n",
    "\n",
    "# Calculate the number of steps for warmup\n",
    "total_steps, warmup_steps = calc_train_steps(\n",
    "    num_example=len(train_sentences),\n",
    "    batch_size=BATCH_SIZE,\n",
    "    epochs=EPOCHS,\n",
    "    warmup_proportion=0.1,\n",
    ")\n",
    "\n",
    "optimizer = AdamWarmup(\n",
    "    total_steps,\n",
    "    warmup_steps,\n",
    "    lr=LEARNING_RATE,\n",
    "    epsilon=1e-6,\n",
    "    weight_decay=0.01,\n",
    "    weight_decay_pattern=['embeddings', 'kernel', 'W1', 'W2', 'Wk', 'Wq', 'Wv', 'Wo']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation metric\n",
    "\n",
    "We'll define a modification of the `categorical_accuracy` metric that ignores the `NO_LABEL` values in the gold data, thus only focusing on predictions for the \"real\" labels.\n",
    "\n",
    "The implementation is a modification of https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/python/keras/metrics.py#L2967 ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.python.ops import math_ops as o\n",
    "from tensorflow.python.keras import backend as K\n",
    "\n",
    "\n",
    "def label_categorical_accuracy(y_true, y_pred):\n",
    "    # Map one-hot targets and predictions to tag indices \n",
    "    y_true_idx = o.argmax(y_true, axis=-1)\n",
    "    y_pred_idx = o.argmax(y_pred, axis=-1)\n",
    "    # Compare targets to predicted elementwise and cast the\n",
    "    # resulting boolean values to floating point values.\n",
    "    # (K.floatx() returns current default float type.)\n",
    "    correct_preds = o.cast(o.equal(y_true_idx, y_pred_idx), K.floatx())\n",
    "    # Compare targets to the special NO_LABEL value and cast\n",
    "    # the resulting boolean values to floating point values.\n",
    "    # This gives a value of zero for NO_LABEL and one for others.\n",
    "    is_label = o.cast(o.not_equal(y_true_idx, tag_to_int[NO_LABEL]), K.floatx())\n",
    "    # Take elementwise product of the comparisons, giving values that\n",
    "    # are one if the prediction is equal to the target and the target\n",
    "    # is not the NO_LABEL value, and zero otherwise.\n",
    "    correct_label_preds = o.multiply(correct_preds, is_label)\n",
    "    # Accuracy is then the number of correct predictions for labels\n",
    "    # divided by the number of labels.\n",
    "    return o.reduce_sum(correct_label_preds)/o.reduce_sum(is_label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train model\n",
    "\n",
    "The model is compiled and trained normally. As our targets are one-hot, we'll use `categorical_crossentropy` loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(\n",
    "    optimizer=optimizer,\n",
    "    loss='categorical_crossentropy',\n",
    "    metrics=[label_categorical_accuracy]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training, as usual. (Note: this will take a fair bit of time unless you're running with GPU acceleration.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/smp/Library/Python/3.7/lib/python/site-packages/tensorflow_core/python/ops/math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
      "Train on 2000 samples, validate on 2000 samples\n",
      "Epoch 1/3\n",
      "2000/2000 [==============================] - 957s 478ms/sample - loss: 0.6913 - label_categorical_accuracy: 0.6605 - val_loss: 0.1031 - val_label_categorical_accuracy: 0.9337\n",
      "Epoch 2/3\n",
      "2000/2000 [==============================] - 932s 466ms/sample - loss: 0.0853 - label_categorical_accuracy: 0.9538 - val_loss: 0.0818 - val_label_categorical_accuracy: 0.9437\n",
      "Epoch 3/3\n",
      "2000/2000 [==============================] - 979s 490ms/sample - loss: 0.0496 - label_categorical_accuracy: 0.9738 - val_loss: 0.0785 - val_label_categorical_accuracy: 0.9482\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(\n",
    "    train_X,\n",
    "    train_Y,\n",
    "    epochs=EPOCHS,\n",
    "    batch_size=BATCH_SIZE,\n",
    "    validation_data=(dev_X, dev_Y)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot that training history:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXhU5f3//+c7k4QQdkgUBRG0IEQhLClQQAGRTRG0KhJEBEURt59YabXyVaDa0rpbLa6o+IGwSkUEEQXqgiABBATKKkKAYggQliQkM7l/f5yTMAmTZAIzc5LJ+3FduThzn2XeczK85uQ+Z+4jxhiUUkqFrwinC1BKKRVcGvRKKRXmNOiVUirMadArpVSY06BXSqkwF+l0AcXFxcWZpk2bOl2GUkpVKmvXrj1sjIn3Na/CBX3Tpk1JTU11ugyllKpUROSXkuZp141SSoU5DXqllApzGvRKKRXmNOiVUirMadArpVSY06BXSqkwp0GvlFJhrsJdR6+UUmHNGDh9ArIOw6kM+990OHUYqteDpJEBf0oNeqWUOh/GwOnjVlBnZVj/nkr3HeQF8z2nfW+rcUcNeqWUCjrv4D512HdQFw9yT67vbUXVgBoNIDYOal0EDVtDbAOoEQc14q32gvk14iC6RlBekga9Uiq8GQM5mV6h7R3eGWfavOfn5/neVnTNM0FduxE0TPQK6nirvWB+bBxEx4b2tZZAg14pVbkYAznHzg7ns4L88Jmj8BKDu9aZoK7TGC5OPHN07euIO6p6aF9rgGjQK6WclZ9vBXdWhlcXiY/wLpjOyoB8t+9tVat95oi6ziVwcbszR9c14opOx8ZBVExoX6tDNOiVUoFVENxF+rIP+why+3FWBhiP721Vq3PmiLrepdC4Q9GgLh7ekdVC+1orCQ16pVTp8vMh+2ixk5LeQV0syLOOlBHcdijXawqNk0o54m6gwR0gGvRKVTX5Hiu4fV5R4uPqkuwjYPJ9byumzpm+7PqXQePfep2UtPu3C+bHNoDI6NC+VgVo0CtV+eV7rKPowr7s9DNB7evqklKDu+6ZoG5wOTTp5HVSsuCKEq+rS1xRoX2t6pxo0CtV0XjcVhiXeFKyWJBnHQGM721Vr3fm0r+45tDkd2dfBlh4xF1fgztMadArFWwetxXMPi/98zGdfRTfwS1WcBeEc/wVUKOr1zXcDYpeGli9Prj0v7jSoFeq/Dx5xbpGSruGuyC4fRHrKLogqC9oVewabq8v3tSIt0Jeg1udA7/eNSLSD3gVcAHvGmMmF5t/KTAViAeOAMOMMWn2PA+wyV50rzFmYIBqVyp08nJg3j2w51vr0kFfJMI6ii4I6guvLPmKkoLgjnCF9nWoKqnMoBcRF/AG0BtIA9aIyAJjzBavxV4AphljPhSRa4G/AXfa87KNMW0DXLdSoWMMfPoI/HchtLvT+gZlkZOSBcFdV4NbVUj+HNF3BHYaY3YDiMhMYBDgHfQJwGP29HLg34EsUilHrXwNNs6Cnk9B9z86XY1S5eZP0DcC9nk9TgM6FVtmA/B7rO6dm4FaItLAGJMBxIhIKuAGJhtjzvoQEJH7gPsAmjRpUu4XoVTQbF8CS5+BhJvgmnFOV6MqmPx8Q15+Pm6PsX7y83HnG/I8dlu+3eax2jz5hjyPj7Z8g9uTT93YKK5teWHA6wzUmZ3HgddFZATwNbAfKPhq3KXGmP0ichmwTEQ2GWN2ea9sjHkbeBsgKSmphOvElAqx9G0w9x5raNmbpoCI0xVVWsZYAecpEoxWKLo9Z9oKgs+dn28FolcoFrR5CoLUDscSt1s8cD1nQriswPUO7aIhXbRmE+C0SrykrmNBvx+4xOtxY7utkDHmANYRPSJSE7jFGHPMnrff/ne3iKwA2gFFgl6pCifrCMy43RqtMDklZMPNGmPsoPPvqLAg5Nye/MKQKmwrDMv8YkFn8BTMyy8ahEWDztc2yg7cs9bJt9pDKTJCiHQJURERRLoEV0QEUa4S2iKESFcE0ZERxLoiiLLXjbSXi/RaNzIionD5KJfgihCiXGfazn7eM/OjXL7brMfWtqtHB+ccjz9BvwZoLiLNsAJ+CDDUewERiQOOGGPygSexrsBBROoBWcaY0/YyXYF/BLB+pc6JMYZcTz45uflk5bnJzvWQleshO89DTk4OrZbdTb3MNL7qOJW9G3PJzt1Bdp6HXHfJR4WFYezjqLAg+LyPPAsC1/uI1h3iQHRFiM/AifQKQF/zY6JKXyeqWPAVhGXxcC3a5nsb3s9hBa6v7Z4JT1eEIPrXVxFlBr0xxi0iDwFLsC6vnGqM2Swik4BUY8wCoAfwNxExWF03D9qrtwLeEpF8rBuRTy52tY5SPhUEcbYdvlm5Hh/T7jPTuR6y8nxMey9jr5tjzy/pKPOZyA+5OvJ7Hs8bzdwVLmArYB0lRkcWDz6vcCoIKq+jwtjIyBIDqcgRYClHhaUFbsG6ZwduhO9t2G0FAR8RoYFYFYgJdCfTeUpKSjKpqalOl6HK4B3EBUfCRafdRUPZZxC7SwnxkoO4JFEuISbKRWy0i9joSK9pl8/p6lEuqkdHek27uGzvHJqvHs/h1vdyovvEwvbYaBdRrogg7U2lzp+IrDXGJPmap1+zC1PGGE6788mxwzMr1+M17fbZfias3WTn5ZOd6y4hxM89iKtHWSFc3Stca1aLJL5mtcJArR4VSfXoCGs5r6AtOh1ZOF0Q3OcdxHu+gzUT4PJexN00mTj9FqoKE/pOdkhBEPvumnCX2GWRY88vGty+pt2Ut7s32hVBTJQVsN4BWhDEsdHWEXD1giNiO3yLTlshXT0q8ky7Pa9CHxEf/QVm32mNkX7rVB1qQIUVfTeXwDuIi3Q52N0NpXVZFD/yPbtrwtrGuQRx8aPb6lEuasVEcmHtaj67IopOn32EHBvtIibaRWyUi8iKHMTBdPokzBxqDT6WPNP6hqtSYSRsgj7Xnc+2/52wjnbz7BNudkifmXb7bC8M6zw32bn55x7EkRE+j3ZrV4/iwtrVinRZ+O43jjyr3TvQq2wQB1N+PswfDb9ugTvmWEP5KhVmwiboM7PzuPH1b0tdJjoywgrQKPso1g7Q2tWjaFg7prCbITbK5WM6smh7sRDWIK6k/jPZGsOm71/hN9c5XY1SQRE2QV83Nop3hyed6YqIdhEbFUmM10k9l15Kprxtng//+Tu0HQadH3C6GqWCJmyCPsoVwXUJgf/qsApTBzfA/DFwSScY8JIOb6DCmvY1qKrn5K+QMtQaavj2/4PIak5XpFRQhc0RvVJ+cZ+GmXdYd4i6ZwnUvMDpipQKOg16VXUYAwvHQtoPcNsHcFGi0xUpFRLadaOqjlX/gh+nQ/c/wZU3O12NUiGjQa+qhp1fwhfjoeUA6P6E09UoFVIa9Cr8Hd4Bc+6GCxLg5rcgQt/2qmrRd7wKb9nHIGWINXbNkBlQrabTFSkVcnoyVoUvjxvm3g1H98Bdn0K9S52uSClHaNCr8PXlM7DrK7jxVbi0i9PVKOUY7bpR4Wn9dPj+deg4GjqMcLoapRylQa/Cz97VsPBRaNbdGqxMqSpOg16Fl8w0mDUM6jS2vhSlNxBRSvvoVRjJzYKUZMjLhhELIba+0xUpVSFo0KvwYAx88gD8bxMMnQ3xVzhdkVIVhga9Cg9fP2+NL997ErTo43Q1SlUo2kevKr+tn8Ly56DNEOjyiNPVKFXh+BX0ItJPRLaJyE4ROWugEBG5VES+EpGNIrJCRBp7zbtLRHbYP3cFsnil+N9P8PFoaNTBul5ebyCi1FnKDHoRcQFvAP2BBCBZRBKKLfYCMM0Y0waYBPzNXrc+8AzQCegIPCMi9QJXvqrSTh22Tr7G1LaGN4iKcboipSokf47oOwI7jTG7jTG5wExgULFlEoBl9vRyr/l9gaXGmCPGmKPAUqDf+Zetqjx3Lsy6E079CkOmQ62GTlekVIXlT9A3AvZ5PU6z27xtAH5vT98M1BKRBn6ui4jcJyKpIpKanp7ub+2qqjIGFj0Oe1fCwNetbhulVIkCdTL2caC7iKwHugP7AY+/Kxtj3jbGJBljkuLj4wNUkgpbP7wD6z6Ebo9Bm9ucrkapCs+fyyv3A5d4PW5stxUyxhzAPqIXkZrALcaYYyKyH+hRbN0V51Gvqup2r4DPn4AW/eHa/+d0NUpVCv4c0a8BmotIMxGJBoYAC7wXEJE4ESnY1pPAVHt6CdBHROrZJ2H72G1KlV/GLph9F8S1gFve0RuIKOWnMv+nGGPcwENYAb0VmG2M2Swik0RkoL1YD2CbiGwHLgSes9c9AvwF68NiDTDJblOqfHKOW1fYiEByClSr5XRFSlUaYoxxuoYikpKSTGpqqtNlqIok32OF/K6v4M750OwapytSqsIRkbXGmCRf83QIBFXxfTURdiyBG17UkFfqHGgnp6rYNsyC716FpLvht6OcrkapSkmDXlVcaamw4GFoejX0/4fT1ShVaWnQq4rp+AGYeYf1jdfbPgRXlNMVKVVpaR+9qnjysmHmUMg9aZ18rdHA6YqUqtQ06FXFYozVXXPgR2sMmwuLj5+nlCov7bpRFcu3L8OmOXDteGh5g9PVKBUWNOhVxbFtMXw1Ca66Ba7+g9PVKBU2NOhVxfDrVpg3Ci5KtEak1BuIKBUwGvTKeVlHIGUIRNewbiASHet0RUqFFT0Zq5zlyYPZw+H4QRjxGdQ563YFSqnzpEGvnPX5k7DnG7jpTbjkt05Xo1RY0q4b5ZzUqbDmHejyMLRNdroapcKWBr1yxs/fwKJx8JvecN1Ep6tRKqxp0KvQO7rH6pevfxnc+h5EuJyuSKmwpkGvQuv0CWtseeOB5JkQU8fpipQKe3oyVoVOfj58PBrSt8GwudDgcqcrUqpK0KBXobP8Odj2GfT7O1x+rdPVKFVlaNeNCo2f5sE3L0C7O6HTaKerUapK0aBXwXdgPfz7AWjyO7jhJR3eQKkQ06BXwXXiEKQMhRrxMPgjiIx2uiKlqhzto1fBk5cDs+6AnGNw9xKoGe90RUpVSX4d0YtIPxHZJiI7ReQJH/ObiMhyEVkvIhtF5Hq7vamIZIvIj/bPm4F+AaqCMgYWjoW0NXDzm3BRG6crUqrKKvOIXkRcwBtAbyANWCMiC4wxW7wWGw/MNsZMEZEEYBHQ1J63yxjTNrBlqwrv+9dhwwzo8SQkDHK6GqWqNH+O6DsCO40xu40xucBMoPj/XAPUtqfrAAcCV6KqdHYshaVPWwF/zR+drkapKs+foG8E7PN6nGa3eZsADBORNKyj+Ye95jWzu3T+IyJX+3oCEblPRFJFJDU9Pd3/6lXFk74d5t4NF14JN02BCD3fr5TTAvW/MBn4wBjTGLge+EhEIoCDQBNjTDvgMWCGiNQuvrIx5m1jTJIxJik+Xk/YVVrZR60biLiiYUiKdSMRpZTj/An6/cAlXo8b223e7gFmAxhjvgdigDhjzGljTIbdvhbYBbQ436JVBeRxw5yRcGwvDJkOdS8pex2lVEj4E/RrgOYi0kxEooEhwIJiy+wFegGISCusoE8XkXj7ZC4ichnQHNgdqOJVBfLFeNi9HAa8DE06O12NUspLmVfdGGPcIvIQsARwAVONMZtFZBKQaoxZAPwBeEdExmKdmB1hjDEicg0wSUTygHzgfmPMkaC9GuWMddNg9RToNAba3+l0NUqpYsQY43QNRSQlJZnU1FSny1D++uV7+PBGaNoN7pgLLv0OnlJOEJG1xpgkX/P0kgh17o7tg1nDoG4TuO19DXmlKigNenVuck9ZNxDx5Fk3EKlez+mKlFIl0EMwVX75+TD/fvh1MwydDfF6IZVSFZkGvSq/r/8BWxdAn2eheW+nq1FKlUG7blT5bPkEVvwNEofC7x5yuhqllB806JX/Dm60umwa/9a6Xl5vIKJUpaBBr/xzMh1mDrVOut4+HaJinK5IKeUn7aNXZXPnWpdRnjoMdy+GWhc6XZFSqhw06FXpjIHPHoN9q+DWqXBxO6crUkqVk3bdqNKtfgvWfwRXPw5X3eJ0NUqpc6BBr0q2axkseRJaDoCeTzldjVLqHGnQK98ydsGcERDfCm5+S28golQlpv971dlyMq0biEREQvIMqFbT6YqUUudBT8aqovI9MPceOLIbhn8C9Zo6XZFS6jxp0KuivnwGdi61vhDVtJvT1SilAkC7btQZP6bAyn/Cb++FpLudrkYpFSAa9Mqybw18+gg0vRr6/c3papRSAaRBryBzP8y6A2pfDIOngSvK6YqUUgGkffRVXW6WNYZNbpZ18jW2vtMVKaUCTIO+KjMGFjwEBzdAcgpc0MrpipRSQaBdN1XZNy/CT/Og19NwRX+nq1FKBYkGfVX1389g2V+g9WDoNtbpapRSQaRBXxUd2gwf3wcXt4eBr+kNRJQKc34FvYj0E5FtIrJTRJ7wMb+JiCwXkfUislFErvea96S93jYR6RvI4tU5OJVhDW8QXROGTIeo6k5XpJQKsjJPxoqIC3gD6A2kAWtEZIExZovXYuOB2caYKSKSACwCmtrTQ4ArgYuBL0WkhTHGE+gXovzgyYPZw+HEIRi52LqcUikV9vw5ou8I7DTG7DbG5AIzgUHFljFAbXu6DnDAnh4EzDTGnDbG/AzstLennLD4j/DLtzDodWjcwelqlFIh4k/QNwL2eT1Os9u8TQCGiUga1tH8w+VYFxG5T0RSRSQ1PT3dz9JVuax5F1KnQtdHoc1gp6tRSoVQoE7GJgMfGGMaA9cDH4mI39s2xrxtjEkyxiTFx8cHqCRV6OevYdEfoXlf61JKpVSV4s8XpvYDl3g9bmy3ebsH6AdgjPleRGKAOD/XVcF05GerX77Bb+CWdyHC5XRFSqkQ8+eoew3QXESaiUg01snVBcWW2Qv0AhCRVkAMkG4vN0REqolIM6A58EOgildlyDkOKcnWN2CTUyCmdtnrKKXCTplH9MYYt4g8BCwBXMBUY8xmEZkEpBpjFgB/AN4RkbFYJ2ZHGGMMsFlEZgNbADfwoF5xEyL5+da18oe3w50fQ4PLna5IKeUQsfK44khKSjKpqalOl1H5fTkRvn0J+j8Pne5zuhqlVJCJyFpjTJKvefrN2HC0cY4V8h1GQMd7na5GKeUwDfpws3+tNSLlpV2to3kd3kCpKk+DPpwcPwgz74CaF1g3EImMdroipVQFoOPRh4u8HOsuUTnH4Z4voEac0xUppSoIDfpwYIx1v9f9a+H2/4OGVzldkVKqAtGum3Cw8jXYOAt6jodWNzpdjVKqgtGgr+y2L4Glz8CVN8M1jztdjVKqAtKgr8zSt8Hce6Bhaxj0L73CRinlkwZ9ZZV1BGbcbt04JDkFomOdrkgpVUHpydjKyOOGOSPg+H4Y8RnUaex0RUqpCkyDvjJa8mf4+T9Wd80leh8XpVTptOumsln7AfzwFvzuIWh3h9PVKKUqAQ36ymTPd/DZH+A310HvSU5Xo5SqJDToK4ujv8DsO6FeM7jlPb2BiFLKbxr0lcHpkzBzqHUSNnkmVK/rdEVKqUpET8ZWdPn5MH80/LoF7pgLcb9xuiKlVCWjQV/R/Wcy/Hch9P0b/KaX09UopSoh7bqpyDbPh//8HdoOg85jnK5GKVVJadBXVAc3wPwxcEknGPCSDm+glDpnGvQV0clfIWUoxDawhh2OrOZ0RUqpSkz76Csa92mYNQyyMuCeJdbdopRS6jxo0FckxsDCx2DfarjtA7go0emKlFJhwK+uGxHpJyLbRGSniDzhY/7LIvKj/bNdRI55zfN4zVsQyOLDzqop8OP/Qfc/WePLK6VUAJR5RC8iLuANoDeQBqwRkQXGmC0Fyxhjxnot/zDQzmsT2caYtoErOUzt/BK+eMq6Q1T3sz5LlVLqnPlzRN8R2GmM2W2MyQVmAoNKWT4ZSAlEcVXG4R0w5264IAFuehMi9By5Uipw/EmURsA+r8dpdttZRORSoBmwzKs5RkRSRWSViNxUwnr32cukpqen+1l6mMg+BilDwBVl3UCkWk2nK1JKhZlAHzoOAeYaYzxebZcaY5KAocArInJ58ZWMMW8bY5KMMUnx8fEBLqkC87hh7t3WgGW3fwR1mzhdkVIqDPkT9PuBS7weN7bbfBlCsW4bY8x++9/dwAqK9t9XbV8+A7u+ghtehEu7OF2NUipM+RP0a4DmItJMRKKxwvysq2dEpCVQD/jeq62eiFSzp+OArsCW4utWSeunw/evQ8fR0OEup6tRSoWxMq+6Mca4ReQhYAngAqYaYzaLyCQg1RhTEPpDgJnGGOO1eivgLRHJx/pQmex9tU6VtXc1LHwULusBff/qdDVKqTAnRXPZeUlJSSY1NdXpMoInMw3e7mmddB31FcTWd7oipVQYEJG19vnQs+g3Y0MpNwtSkiEvG0Ys1JBXSoWEBn2oGAOfPAD/2wRDZ0P8FU5XpJSqIjToQ+XrF6zx5XtPghZ9nK5GKVWF6FcwQ2Hrp7D8WWgzBLo84nQ1SqkqRoM+2P73E3w8GholwY2v6g1ElFIhp0EfTKcOWydfY2rDkOkQFeN0RUqpKkj76IPFnQuz7oRTv8LIxVCrodMVKaWqKA36YDAGFj0Oe1fCLe9Bo/ZOV6SUqsK06yYYfngH1n0I3R6D1rc6XY1SqorToA+03Svg8yegRX+49v85XY1SSmnQB1TGLph9F8S1gFve0RuIKKUqBE2iQMk5bl1hI2LfQKSW0xUppRSgJ2MDI98D80bBkV1w53yo38zpipRSqpAGfSB8NQl2LLFuINLsGqerUUqpIrTr5nxtmAXfvQJJ98BvRzldjVJKnUWD/nykrYUFD0PTq6H/352uRimlfNKgP1fHD8DModY3Xm/7EFxRTleklFI+aR/9ucjLtkI+96R18rVGA6crUkqpEmnQl5cxVnfNgR9hyAy4MMHpipRSqlTadVNe374Mm+bAteOh5fVOV6OUUmXSoC+PbYutSymvugWu/oPT1SillF806P3161brS1EXJcLA1/UGIkqpSkOD3h9ZRyBlCETXsPrlo2Odrkgppfzm18lYEekHvAq4gHeNMZOLzX8Z6Gk/jAUuMMbUtefdBYy35z1rjPkwEIWHjCcPZg+H4wdhxGdQp5HTFakKLi8vj7S0NHJycpwuRYWhmJgYGjduTFSU/5d0lxn0IuIC3gB6A2nAGhFZYIzZUrCMMWas1/IPA+3s6frAM0ASYIC19rpH/a7QaZ8/CXu+gZvfgkt+63Q1qhJIS0ujVq1aNG3aFNEuPhVAxhgyMjJIS0ujWTP/x9Typ+umI7DTGLPbGJMLzAQGlbJ8MpBiT/cFlhpjjtjhvhTo53d1TkudCmvegS6PQOIQp6tRlUROTg4NGjTQkFcBJyI0aNCg3H8t+hP0jYB9Xo/T7DZfRVwKNAOWlWddEblPRFJFJDU9Pd2fuoNvz7ewaBw07wPXTXC6GlXJaMirYDmX91agT8YOAeYaYzzlWckY87YxJskYkxQfHx/gks7B0T3Wjb3rXwa3vAsRLqcrUkqpc+ZP0O8HLvF63Nhu82UIZ7ptyrtuxXD6hHUDEZMPyTMhpo7TFSlVLj179mTJkiVF2l555RXGjBlT6no1a9YE4MCBA9x6q+97Hffo0YPU1NRSt/PKK6+QlZVV+Pj666/n2LFj/pQeMHv27GHGjBkhfc6KzJ+gXwM0F5FmIhKNFeYLii8kIi2BesD3Xs1LgD4iUk9E6gF97LaKKT8fPh4N6dvgtg+gweVOV6RUuSUnJzNz5swibTNnziQ5Odmv9S+++GLmzp17zs9fPOgXLVpE3bp1z3l756KiBL3b7Xa6BMCPq26MMW4ReQgroF3AVGPMZhGZBKQaYwpCfwgw0xhjvNY9IiJ/wfqwAJhkjDkS2JcQQMufg22fQb+/w+U9y15eqTJM/HQzWw4cD+g2Ey6uzTM3Xlni/FtvvZXx48eTm5tLdHQ0e/bs4cCBA1x99dWcPHmSQYMGcfToUfLy8nj22WcZNKjotRV79uxhwIAB/PTTT2RnZzNy5Eg2bNhAy5Ytyc7OLlxuzJgxrFmzhuzsbG699VYmTpzIa6+9xoEDB+jZsydxcXEsX76cpk2bkpqaSlxcHC+99BJTp04FYNSoUTz66KPs2bOH/v37061bN1auXEmjRo345JNPqF69epG65syZw8SJE3G5XNSpU4evv/4aj8fDE088wYoVKzh9+jQPPvggo0eP5oknnmDr1q20bduWu+66i7FjCy8MLHUfTJs2jRdeeAERoU2bNnz00UccOnSI+++/n927dwMwZcoULr744sJ9BPDCCy9w8uRJJkyYQI8ePWjbti3ffvstycnJtGjRgmeffZbc3FwaNGjA9OnTufDCCzl58iQPP/wwqampiAjPPPMMmZmZbNy4kVdeeQWAd955hy1btvDyyy+f69sF8PM6emPMImBRsbaniz2eUMK6U4Gp51hf6Pw0D755AdoPh06jna5GqXNWv359OnbsyOLFixk0aBAzZ85k8ODBiAgxMTHMnz+f2rVrc/jwYTp37szAgQNLPME3ZcoUYmNj2bp1Kxs3bqR9+/aF85577jnq16+Px+OhV69ebNy4kUceeYSXXnqJ5cuXExcXV2Rba9eu5f3332f16tUYY+jUqRPdu3enXr167Nixg5SUFN555x0GDx7MvHnzGDZsWJH1J02axJIlS2jUqFFhV9B7771HnTp1WLNmDadPn6Zr16706dOHyZMn88ILL7Bw4cKzXlNJ+2DLli08++yzrFy5kri4OI4csY5JH3nkEbp37878+fPxeDycPHmSo0dLv0I8Nze3sIvr6NGjrFq1ChHh3Xff5R//+Acvvvgif/nLX6hTpw6bNm0qXC4qKornnnuO559/nqioKN5//33eeuutUp/LHzp6JcCB9fDvB6DJ7+D6F3V4AxUwpR15B1NB901B0L/33nuAdR32n//8Z77++msiIiLYv38/hw4domHDhj638/XXX/PII48A0KZNG9q0aVM4b/bs2bz99tu43W4OHjzIli1biswv7ttvv+Xmm2+mRo0aAPz+97/nm2++YeDAgTRr1oy2bdsC0KFDB/bs2XPW+l27dmXEiBEMHjyY3//+9wB88bAmUE4AABI3SURBVMUXbNy4sbCrKTMzkx07dhAdHV1iHSXtg2XLlnHbbbcVfkDVr18fgGXLljFt2jSAwr8mygr622+/vXA6LS2N22+/nYMHD5Kbm1t4/fuXX35ZpIutXr16AFx77bUsXLiQVq1akZeXR+vWrUt9Ln9o0J84BClDoUY8DP4IIkt+gyhVWQwaNIixY8eybt06srKy6NChAwDTp08nPT2dtWvXEhUVRdOmTc/pG7w///wzL7zwAmvWrKFevXqMGDHivL4JXK1atcJpl8tVpIuowJtvvsnq1av57LPP6NChA2vXrsUYwz//+U/69u1bZNkVK1aU+FyB2AeRkZHk5+cXPi6+fsGHGcDDDz/MY489xsCBA1mxYgUTJkwoddujRo3ir3/9Ky1btmTkyJHlqqskVXusm7wcmHUH5ByzxrCpWQEu7VQqAGrWrEnPnj25++67i5yEzczM5IILLiAqKorly5fzyy+/lLqda665pvCk5k8//cTGjRsBOH78ODVq1KBOnTocOnSIxYsXF65Tq1YtTpw4cda2rr76av7973+TlZXFqVOnmD9/PldffbXfr2nXrl106tSJSZMmER8fz759++jbty9TpkwhLy8PgO3bt3Pq1KkSayhtH1x77bXMmTOHjIwMgMKum169ejFlyhQAPB4PmZmZXHjhhfz6669kZGRw+vRpn11E3s/XqJH19aEPPzwzAkzv3r154403Ch8X/JXQqVMn9u3bx4wZM/w+gV6Wqhv0xsDCsZC2xhre4KKS/+RUqjJKTk5mw4YNRcLijjvuIDU1ldatWzNt2jRatmxZ6jbGjBnDyZMnadWqFU8//XThXwaJiYm0a9eOli1bMnToULp27Vq4zn333Ue/fv3o2bPoBQ3t27dnxIgRdOzYkU6dOjFq1CjatWvn9+sZN24crVu35qqrrqJLly4kJiYyatQoEhISaN++PVdddRWjR4/G7XbTpk0bXC4XiYmJZ53ILGkfXHnllTz11FN0796dxMREHnvsMQBeffVVli9fTuvWrenQoQNbtmwhKiqKp59+mo4dO9K7d+9S9+OECRO47bbb6NChQ5HzFuPHj+fo0aNcddVVJCYmsnz58sJ5gwcPpmvXroXdOedLvC6SqRCSkpJMWdfpBsTKf8IX46HHn6HHn4L/fKrK2Lp1K61atXK6DFWJDRgwgLFjx9KrVy+f8329x0RkrTEmydfyVfOIfsdSWPo0JAyCa8Y5XY1SSgFw7NgxWrRoQfXq1UsM+XNR9U7Gpm+HuXfDhVfCTVMgomp+1imlKp66deuyffv2gG+3aqVc9lHrBiKR1WBIinUjEaWUCnNV54je44Y5I+HYXhixEOpeUvY6SikVBqpO0H8xHnYvt+732qSz09UopVTIVI2um3XTYPUU6PwAtL/T6WqUUiqkwj/of/keFj4Gl/WE3n9xuhqlgi4jI4O2bdvStm1bGjZsSKNGjQof5+bm+rWNkSNHsm3btlKXeeONN5g+fXogSi6XZcuWsWrVqpA/b2UW3l03x/bBrGFQtwnc9j64wvvlKgXQoEEDfvzxR8D6sk7NmjV5/PHHiyxjjMEYQ0QJV529//77ZT7Pgw8+eP7FnoNly5YRFxdH587OdsF6PB5crspxU6LwTb7cU9YNRDx51g1EqgfmG2ZKlcviJ+B/mwK7zYatof/kcq+2c+dOBg4cSLt27Vi/fj1Lly5l4sSJrFu3juzsbG6//XaeftoalLZbt268/vrrXHXVVcTFxXH//fezePFiYmNj+eSTT7jgggsYP348cXFxPProo3Tr1o1u3bqxbNkyMjMzef/99+nSpQunTp1i+PDhbN26lYSEBPbs2cO7775bOIBZgXHjxvHZZ58RGRlJ//79+fvf/86hQ4cYM2YMe/fuJSIigtdee434+HjeffddXC4XH3zwAf/617/o0qVL4XZWrVrF2LFjycnJITY2lg8++IDmzZvjdrsZN24cS5cuJSIigvvvv58HHniA1atX8+ijj5KVlUVMTAzLly9nxowZ/PTTT4VDBffr14/x48fTuXNn4uLiGDFiBMuWLeOtt97i888/Z9GiRWRnZ9OtWzemTJmCiLB9+3buv/9+MjIycLlcfPzxx/z5z39m6NChDBgwALAGPhs+fDg33HDDub4T/BaeQZ+fD/Pvh183w9A5EN/C6YqUqhD++9//Mm3aNJKSrC9QTp48mfr16+N2u+nZsye33norCQkJRdbJzMyke/fuTJ48mccee4ypU6fyxBNPnLVtYww//PADCxYsYNKkSXz++ef885//pGHDhsybN48NGzYUGea4wKFDh1i0aBGbN29GRAqHIH7kkUf44x//SOfOnYuMkT9q1KjCD5jiWrVqxTfffENkZCSff/4548ePZ9asWUyZMoUDBw6wYcMGXC4XR44cIScnhyFDhjBv3jzat29PZmZmkcHVfMnMzOSaa64p/BC44oormDhxIsYYhg4dyueff07//v1JTk5mwoQJ3HjjjeTk5JCfn88999zDlClTGDBgAEePHmXNmjUhuzlKeAb91/+ArQugz3PQ/Dqnq1FV2TkceQfT5ZdfXhjyACkpKbz33nu43W4OHDjAli1bzgr66tWr079/f8AaQvibb77xue2CoYO9hxn+9ttv+dOfrCFGEhMTufLKs4dtrl+/PhEREdx7773ccMMNhUe8X375ZZHzBEePHvU5qqW3Y8eOMXz4cHbt2lWk/csvv+TRRx8t7GqpX78+69evp0mTJoUfPnXqlH3b0OjoaG6++ebCx1999RXPP/88OTk5HD58mA4dOtC5c2cOHz7MjTfeCFjj34M1aNpDDz1ERkYGKSkpDB48OGRdP+EX9Fs+gRV/g7Z3wO+c6UNUqqLyHj53x44dvPrqq/zwww/UrVuXYcOG+Ryu13tsd5fLVeLt8QqOhktbxpeoqChSU1NZunQpc+bMYcqUKXzxxReFfyGUNrZ8cU899RR9+/blgQceYOfOnfTr18/vdQuUNgRx9erVC2/SkpWVxUMPPcS6deto1KgR48ePL3W4YxFh2LBhzJgxgw8//DCkJ7LD66qbgxutLpvGHWHAy3oDEaVKcfz4cWrVqkXt2rU5ePDgWTcUD4SuXbsye/ZsADZt2sSWLVvOWubEiRMcP36cAQMG8PLLL7N+/XoArrvuuiLD+BacYC5rCOKCIYE/+OCDwvbevXvz5ptv4vF4AGsI4oSEBPbu3cu6desAa394PB6aNm3K+vXrMcawZ88e1q5d6/O5srOziYiIIC4ujhMnTjBv3jzAuoFIfHw8n376KWB9UBTcQ3fkyJE8//zzVKtWjSuuuMKPPRgY4RP0J9Nh5lDrpOvt/2cNc6CUKlH79u1JSEigZcuWDB8+vMhQw4Hy8MMPs3//fhISEpg4cSIJCQlndZFkZmZyww03kJiYSPfu3XnppZcA6/LN7777jjZt2pCQkMA777wDWDdVmT17Nu3atWPlypVFtvWnP/2JcePG0b59e7xH5h09ejQNGzakTZs2JCYmMnv2bKpVq0ZKSgpjxowhMTGRPn36cPr0abp3706jRo1o1aoVf/jDH846cVygQYMG3HXXXSQkJNC/f386depUOG/69Om8+OKLtGnThm7dupGeng5YN15v0aJFwG4o4q/wGaY464h1O8AeT8DFvn8xSoWCDlN8htvtxu12ExMTw44dO+jTpw87duwgMjL8eo39cerUKVq3bs2GDRuoVavWOW+nvMMUh8/ejq0PQ2eWvZxSKmROnjxJr169cLvdGGN46623qmzIL1myhHvvvZdx48adV8ifi6q5x5VSIVG3bt0S+7irmr59+7J3715HntuvPnoR6Sci20Rkp4icfQGttcxgEdkiIptFZIZXu0dEfrR/FgSqcKUqsorWJarCx7m8t8o8ohcRF/AG0BtIA9aIyAJjzBavZZoDTwJdjTFHReQCr01kG2O001xVGTExMWRkZNCgQYPCS/GUCgRjDBkZGYXX5vvLn66bjsBOY8xuABGZCQwCvK+Tuhd4wxhz1C7m13JVoVQYady4MWlpaYVXWigVSDExMTRu3Lhc6/gT9I2AfV6P04BOxZZpASAi3wEuYIIx5vOCukQkFXADk40x/y5XhUpVMlFRUTRr1szpMpQqFKiTsZFAc6AH0Bj4WkRaG2OOAZcaY/aLyGXAMhHZZIwp8v1kEbkPuA+gSZMmASpJKaUU+Hcydj/gfd+9xnabtzRggTEmzxjzM7AdK/gxxuy3/90NrADaFX8CY8zbxpgkY0xSfHx8uV+EUkqpkvkT9GuA5iLSTESigSFA8atn/o11NI+IxGF15ewWkXoiUs2rvStF+/aVUkoFWZldN8YYt4g8BCzB6n+faozZLCKTgFRjzAJ7Xh8R2QJ4gHHGmAwR6QK8JSL5WB8qk72v1vFl7dq1h0Xkl/N4TXHA4fNYP1i0rvLRuspH6yqfcKzr0pJmVLghEM6XiKSW9DVgJ2ld5aN1lY/WVT5Vra7wGdRMKaWUTxr0SikV5sIx6N92uoASaF3lo3WVj9ZVPlWqrrDro1dKKVVUOB7RK6WU8qJBr5RSYa7SBH1ZQyWLSDURmWXPXy0iTb3mPWm3bxORviGu6zF7+OaNIvKViFzqNS9oQzj7UdcIEUn3ev5RXvPuEpEd9s9dIa7rZa+atovIMa95wdxfU0XkVxH5qYT5IiKv2XVvFJH2XvOCub/KqusOu55NIrJSRBK95u2x23+0x5sKZV09RCTT6/f1tNe8Moc9D2Jd47xq+sl+T9W35wVzf10iIsvlzFDu/5+PZYL3HjPGVPgfrC9q7QIuA6KBDUBCsWUeAN60p4cAs+zpBHv5akAzezuuENbVE4i1p8cU1GU/Pung/hoBvO5j3frAbvvfevZ0vVDVVWz5h7G+oBfU/WVv+xqgPfBTCfOvBxYDAnQGVgd7f/lZV5eC5wP6F9RlP94DxDm0v3oAC8/3PRDouooteyOwLET76yKgvT1dC2uYmOL/J4P2HqssR/SFQyUbY3KBgqGSvQ0CPrSn5wK9RETs9pnGmNPGGodnp729kNRljFlujMmyH67CGiso2PzZXyXpCyw1xhwx1rDTS4F+DtWVDKQE6LlLZYz5GjhSyiKDgGnGsgqoKyIXEdz9VWZdxpiV9vNC6N5f/uyvkpzPezPQdYXy/XXQGLPOnj4BbMUaGdhb0N5jlSXofQ2VXHwnFS5jjHEDmUADP9cNZl3e7sH6xC4QIyKpIrJKRG4KUE3lqesW+0/EuSJSMHBdhdhfdhdXM2CZV3Ow9pc/Sqo9mPurvIq/vwzwhYisFWuE2FD7nYhsEJHFInKl3VYh9peIxGKF5Tyv5pDsL7G6ldsBq4vNCtp7TO8ZGyIiMgxIArp7NV9qyhjCOYg+BVKMMadFZDTWX0PXhui5/TEEmGuM8Xi1Obm/KjQR6YkV9N28mrvZ++sCYKmI/Nc+4g2FdVi/r5Micj3WwIfNQ/Tc/rgR+M4Y4330H/T9JSI1sT5cHjXGHA/ktktTWY7o/RkquXAZEYkE6gAZfq4bzLoQkeuAp4CBxpjTBe3GjyGcg1WXMSbDq5Z3gQ7+rhvMurwModif1UHcX/4oqfZg7i+/iEgbrN/hIGNMRkG71/76FZhP4Losy2SMOW6MOWlPLwKixBrB1vH9ZSvt/RWU/SUiUVghP90Y87GPRYL3HgvGiYdA/2D95bEb60/5ghM4VxZb5kGKnoydbU9fSdGTsbsJ3MlYf+pqh3XyqXmx9npANXs6DthBgE5K+VnXRV7TNwOrzJkTPz/b9dWzp+uHqi57uZZYJ8YkFPvL6zmaUvLJxRsoeqLsh2DvLz/raoJ13qlLsfYaQC2v6ZVAvxDW1bDg94cVmHvtfefXeyBYddnz62D149cI1f6yX/s04JVSlgnaeyxgOzfYP1hnpLdjheZTdtskrKNkgBhgjv2m/wG4zGvdp+z1tgH9Q1zXl8Ah4Ef7Z4Hd3gXYZL/RNwH3hLiuvwGb7edfDrT0Wvduez/uBEaGsi778QSsIa291wv2/koBDgJ5WH2g9wD3A/fb8wV4w657E5AUov1VVl3vAke93l+pdvtl9r7aYP+enwpxXQ95vb9W4fVB5Os9EKq67GVGYF2g4b1esPdXN6xzABu9flfXh+o9pkMgKKVUmKssffRKKaXOkQa9UkqFOQ16pZQKcxr0SikV5jTolVIqzGnQK6VUmNOgV0qpMPf/AyGRgL6S2nzKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "def plot_history(history):\n",
    "    train_metric = 'label_categorical_accuracy'\n",
    "    val_metric = 'val_label_categorical_accuracy'\n",
    "    plt.plot(history.history[val_metric],label=\"Validation set accuracy\")\n",
    "    plt.plot(history.history[train_metric],label=\"Training set accuracy\")\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "plot_history(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's not a bad result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predict with the trained model\n",
    "\n",
    "Let's run a few simple cases through the trained model to illustrate its use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This\tDET\n",
      "model\tNOUN\n",
      "can\tAUX\n",
      "predict\tVERB\n",
      "parts\tNOUN\n",
      "of\tADP\n",
      "speech\tNOUN\n",
      ".\tPUNCT\n",
      "\n",
      "What\tPRON\n",
      "would\tAUX\n",
      "the\tDET\n",
      "best\tADJ\n",
      "time\tNOUN\n",
      "be\tAUX\n",
      "tomorrow,\tNOUN\n",
      "John\tPROPN\n",
      "wondered\tVERB\n",
      "quietly\tADV\n",
      ".\tPUNCT\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def predict_tags(words):\n",
    "    # This function takes a sequence of words, vectorizes it,\n",
    "    # and returns the model predictions. The vectorization code\n",
    "    # we wrote expects labels, so we'll attach dummy labels to\n",
    "    # use the code and then discard the dummy labels it generates.\n",
    "    dummy_tagged = [(word, NO_LABEL) for word in words]\n",
    "    tokenized_test = tokenize_sentences([dummy_tagged])\n",
    "    test_X, dummy_Y = vectorize_dataset(tokenized_test)\n",
    "    # Run model.predict for this single sequence \n",
    "    predictions = model.predict(test_X)\n",
    "    # Our outputs are one-hot, take argmax to get indices\n",
    "    y = predictions[0].argmax(axis=1)\n",
    "    # For words tokenized into several subword parts, we\n",
    "    # only care about the predicted tag for the first part.\n",
    "    # Use the tokenization to identify these parts.\n",
    "    tags = []\n",
    "    i = 1    # Start at 1 to skip the special [CLS] token.\n",
    "    for tokens, _ in tokenized_test[0]:\n",
    "        tags.append(int_to_tag[y[i]])\n",
    "        i += len(tokens)\n",
    "    return tags\n",
    "\n",
    "\n",
    "# Test the model with a few word sequences\n",
    "test_words = [\n",
    "  'This model can predict parts of speech .'.split(),\n",
    "  'What would the best time be tomorrow, John wondered quietly .'.split()\n",
    "]\n",
    "\n",
    "\n",
    "for words in test_words:\n",
    "    tags = predict_tags(words)\n",
    "    for w, t in zip(words, tags):\n",
    "        print('{}\\t{}'.format(w, t))\n",
    "    print()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}