rnn_named_entity_recognition.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Named entity recognition</h2>\n",
    "<br>\n",
    "Sequence labelling: input word sequence, output tag sequence; inputs and outputs paired (one output for one input)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"ner.png\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Load CoNLL'03 data</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the CoNLL'03 dataset. The data in their original format can be found here https://github.com/glample/tagger .\n",
    "\n",
    "In this demo, the dataset has already been converted to json files. The converted data are found here under the file names `ner_train.json` and `ner_test.json`: https://github.com/TurkuNLP/Deep_Learning_in_LangTech_course/tree/master/data The code used for convertion can be found here: https://github.com/TurkuNLP/Deep_Learning_in_LangTech_course/blob/master/read_ner.ipynb \n",
    "\n",
    "First, download the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File ‘ner-conll03-en-train.json’ already there; not retrieving.\n",
      "\n",
      "File ‘ner-conll03-en-dev.json’ already there; not retrieving.\n",
      "\n",
      "File ‘ner-conll03-en-test.json’ already there; not retrieving.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Give -nc (--no-clobber) argument so that the file isn't downloaded multiple times \n",
    "!wget -nc https://raw.githubusercontent.com/TurkuNLP/Deep_Learning_in_LangTech_course/master/data/ner-conll03-en-train.json\n",
    "!wget -nc https://raw.githubusercontent.com/TurkuNLP/Deep_Learning_in_LangTech_course/master/data/ner-conll03-en-dev.json\n",
    "!wget -nc https://raw.githubusercontent.com/TurkuNLP/Deep_Learning_in_LangTech_course/master/data/ner-conll03-en-test.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is divided into separate sentences and has also been tokenized already. You don't have to do any preprocessing. The produced json files contain a single dictionary for each sentence. The dictionary has a list of tokens and the corresponding list of labels, e.g.: { \"text\": [ \"EU\", \"rejects\", \"German\", \"call\", \"to\", \"boycott\", \"British\", \"lamb\", \".\" ], \"tags\": [ \"I-ORG\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\" ] },"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'list'>\n",
      "<class 'dict'>\n",
      "dict_keys(['text', 'tags'])\n",
      "{'text': ['EU', 'rejects', 'German', 'call', 'to', 'boycott', 'British', 'lamb', '.'], 'tags': ['I-ORG', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "\n",
    "with open(\"ner-conll03-en-train.json\") as f:\n",
    "    data = json.load(f)\n",
    "\n",
    "    \n",
    "# Look at the data\n",
    "print(type(data))\n",
    "print(type(data[0]))\n",
    "print(data[0].keys())\n",
    "print(data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We won't be making use of cross-sentence information, so we can shuffle the data on the sentence level (**not** the token level). Let's also separate the texts and tags from the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text: ['9/16', '-', 'Luo', 'Yigang', '(', 'China', ')', 'beat', 'Jason', 'Wong', '(', 'Malaysia', ')', '15-5', '15-6']\n",
      "Label: ['O', 'O', 'I-PER', 'I-PER', 'O', 'I-LOC', 'O', 'O', 'I-PER', 'I-PER', 'O', 'I-LOC', 'O', 'O', 'O']\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "import numpy\n",
    "\n",
    "\n",
    "random.seed(123)    # This makes the shuffle produce the same order every time\n",
    "random.shuffle(data)\n",
    "\n",
    "train_texts = [example[\"text\"] for example in data]\n",
    "train_labels = [example[\"tags\"] for example in data]\n",
    "\n",
    "# Example text and labels\n",
    "print('Text:', train_texts[0])\n",
    "print('Label:', train_labels[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the development (validation) data. A separate validation set involving different documents than the train data is used as splitting one set of documents into training and validation data would result in overly optimistic evaluation results due to e.g. repeated names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"ner-conll03-en-dev.json\") as f:\n",
    "    validation_data = json.load(f)\n",
    "\n",
    "validation_texts = [example[\"text\"] for example in validation_data]\n",
    "validation_labels = [example[\"tags\"] for example in validation_data]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load pretrained embeddings\n",
    "\n",
    "We'll initialize our embeddings with word vectors pretrained on the English Wikipedia.\n",
    "\n",
    "(Don't worry if the download, unpacking and loading take some time, this is a fair amount of data.)\n",
    "\n",
    "Download embeddings from https://fasttext.cc/docs/en/english-vectors.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File ‘wiki-news-300d-1M.vec.zip’ already there; not retrieving.\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!wget -nc https://dl.fbaipublicfiles.com/fasttext/vectors-english/wiki-news-300d-1M.vec.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  wiki-news-300d-1M.vec.zip\r\n"
     ]
    }
   ],
   "source": [
    "# Give -n argument so that a possible existing file isn't overwritten \n",
    "!unzip -n wiki-news-300d-1M.vec.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use gensim to read the embedding model.\n",
    "\n",
    "Note that we are using a vocabulary size of only 50,000 so that the model trains faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Words from embedding model: 50000\n",
      "First 50 words: [',', 'the', '.', 'and', 'of', 'to', 'in', 'a', '\"', ':', ')', 'that', '(', 'is', 'for', 'on', '*', 'with', 'as', 'it', 'The', 'or', 'was', \"'\", \"'s\", 'by', 'from', 'at', 'I', 'this', 'you', '/', 'are', '=', 'not', '-', 'have', '?', 'be', 'which', ';', 'all', 'his', 'has', 'one', 'their', 'about', 'but', 'an', '|']\n",
      "Before normalization: [-0.0234 -0.0268 -0.0838  0.0386 -0.0321  0.0628  0.0281 -0.0252  0.0269\n",
      " -0.0063]\n",
      "After normalization: [-0.0163762  -0.01875564 -0.05864638  0.02701372 -0.02246478  0.04394979\n",
      "  0.01966543 -0.0176359   0.01882563 -0.00440898]\n"
     ]
    }
   ],
   "source": [
    "from gensim.models import KeyedVectors\n",
    "\n",
    "vector_model = KeyedVectors.load_word2vec_format(\"wiki-news-300d-1M.vec\", binary=False, limit=50000)\n",
    "\n",
    "\n",
    "# sort based on the index to make sure they are in the correct order\n",
    "words = [k for k, v in sorted(vector_model.vocab.items(), key=lambda x: x[1].index)]\n",
    "print(\"Words from embedding model:\", len(words))\n",
    "print(\"First 50 words:\", words[:50])\n",
    "\n",
    "# Normalize the vectors to unit length\n",
    "print(\"Before normalization:\", vector_model.get_vector(\"in\")[:10])\n",
    "vector_model.init_sims(replace=True)\n",
    "print(\"After normalization:\", vector_model.get_vector(\"in\")[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Words in vocabulary: 50002\n",
      "Found pretrained vectors for 50000 words.\n"
     ]
    }
   ],
   "source": [
    "# Build vocabulary mappings\n",
    "\n",
    "# Zero is used for padding in Keras, prevent using it for a normal word.\n",
    "# Also reserve an index for out-of-vocabulary items.\n",
    "vocabulary={\n",
    "    \"<PAD>\": 0,\n",
    "    \"<OOV>\": 1\n",
    "}\n",
    "\n",
    "for word in words: # These are words from the word2vec model\n",
    "    vocabulary.setdefault(word, len(vocabulary))\n",
    "\n",
    "print(\"Words in vocabulary:\",len(vocabulary))\n",
    "inv_vocabulary = { value: key for key, value in vocabulary.items() } # invert the dictionary\n",
    "\n",
    "\n",
    "# Embedding matrix\n",
    "def load_pretrained_embeddings(vocab, embedding_model):\n",
    "    \"\"\" vocab: vocabulary from our data vectorizer, embedding_model: model loaded with gensim \"\"\"\n",
    "    pretrained_embeddings = numpy.random.uniform(low=-0.05, high=0.05, size=(len(vocab)-1,embedding_model.vectors.shape[1]))\n",
    "    pretrained_embeddings = numpy.vstack((numpy.zeros(shape=(1,embedding_model.vectors.shape[1])), pretrained_embeddings))\n",
    "    found=0\n",
    "    for word,idx in vocab.items():\n",
    "        if word in embedding_model.vocab:\n",
    "            pretrained_embeddings[idx]=embedding_model.get_vector(word)\n",
    "            found+=1\n",
    "            \n",
    "    print(\"Found pretrained vectors for {found} words.\".format(found=found))\n",
    "    return pretrained_embeddings\n",
    "\n",
    "pretrained=load_pretrained_embeddings(vocabulary, vector_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Preprocess data</h3>\n",
    "<br>\n",
    "Both the text and the labels are strings; we'll need to convert the labels into integers and the text into an appropriate format for RNN training.<br>\n",
    "<h4>Labels</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'I-LOC': 1, 'I-ORG': 0, 'I-PER': 3, 'O': 2}\n"
     ]
    }
   ],
   "source": [
    "from pprint import pprint\n",
    "\n",
    "\n",
    "# Label mappings\n",
    "# 1) gather a set of unique labels\n",
    "label_set = set()\n",
    "for sentence_labels in train_labels: #loops over sentences \n",
    "    for label in sentence_labels: #loops over labels in one sentence\n",
    "        label_set.add(label)\n",
    "\n",
    "# 2) index these\n",
    "label_map = {}\n",
    "for index, label in enumerate(label_set):\n",
    "    label_map[label]=index\n",
    "    \n",
    "pprint(label_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2, 2, 3, 3, 2, 1, 2, 2, 2, 2, 3, 3, 2, 1, 2, 2, 2, 2]\n"
     ]
    }
   ],
   "source": [
    "# vectorize the labels\n",
    "def label_vectorizer(labels,label_map):\n",
    "    vectorized_labels = []\n",
    "    for label in labels:\n",
    "        vectorized_example_label = []\n",
    "        for token in label:\n",
    "            vectorized_example_label.append(label_map[token])\n",
    "        vectorized_labels.append(vectorized_example_label)\n",
    "    vectorized_labels = numpy.array(vectorized_labels)\n",
    "    return vectorized_labels\n",
    "        \n",
    "\n",
    "vectorized_labels = label_vectorizer(train_labels, label_map)\n",
    "validation_vectorized_labels = label_vectorizer(validation_labels, label_map)\n",
    "\n",
    "pprint(vectorized_labels[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Texts</h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next comes the vectorization of the texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['4', '-', 'Kimiko', 'Date', '(', 'Japan', ')', 'beat', '2', '-', 'Conchita', 'Martinez', '(', 'Spain', ')', '6-2', '7-5', '.']\n",
      "[225, 37, 1, 9859, 14, 799, 12, 2270, 142, 37, 1, 10290, 14, 1366, 12, 13775, 21996, 4]\n",
      "['4', '-', '<OOV>', 'Date', '(', 'Japan', ')', 'beat', '2', '-', '<OOV>', 'Martinez', '(', 'Spain', ')', '6-2', '7-5', '.']\n"
     ]
    }
   ],
   "source": [
    "def text_vectorizer(vocab, texts):\n",
    "    vectorized_data = [] # turn text into numbers based on our vocabulary mapping\n",
    "    sentence_lengths = [] # Number of tokens in each sentence\n",
    "    \n",
    "    for i, one_example in enumerate(texts):\n",
    "        vectorized_example = []\n",
    "        for word in one_example:\n",
    "            vectorized_example.append(vocab.get(word, 1)) # 1 is our index for out-of-vocabulary tokens\n",
    "\n",
    "        vectorized_data.append(vectorized_example)     \n",
    "        sentence_lengths.append(len(one_example))\n",
    "        \n",
    "    vectorized_data = numpy.array(vectorized_data) # turn python list into numpy array\n",
    "    \n",
    "    return vectorized_data, sentence_lengths\n",
    "\n",
    "vectorized_data, lengths=text_vectorizer(vocabulary, train_texts)\n",
    "validation_vectorized_data, validation_lengths=text_vectorizer(vocabulary, validation_texts)\n",
    "\n",
    "print(train_texts[0])\n",
    "print(vectorized_data[0])\n",
    "print([inv_vocabulary[i] for i in vectorized_data[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that there are a lot of out-of-vocabulary tokens because of the small vocabulary we use. The number of `<OOV>` tokens can be decreased by using a larger vocabulary.<br><br>\n",
    "We have to pad the sequences so that all of them are of the same length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Old shape: (14041,)\n",
      "New shape: (14041, 113)\n",
      "First example: [  225    37     1  9859    14   799    12  2270   142    37     1 10290\n",
      "    14  1366    12 13775 21996     4     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0]\n",
      "Padded labels shape: (14041, 113, 1)\n",
      "{'I-LOC': 1, 'I-ORG': 0, 'I-PER': 3, 'O': 2}\n",
      "First example labels:\n",
      "array([[2],\n",
      "       [2],\n",
      "       [3],\n",
      "       [3],\n",
      "       [2],\n",
      "       [1],\n",
      "       [2],\n",
      "       [2],\n",
      "       [2],\n",
      "       [2],\n",
      "       [3],\n",
      "       [3],\n",
      "       [2],\n",
      "       [1],\n",
      "       [2],\n",
      "       [2],\n",
      "       [2],\n",
      "       [2],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0],\n",
      "       [0]], dtype=int32)\n",
      "First weight vector: [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "\n",
    "print(\"Old shape:\", vectorized_data.shape)\n",
    "vectorized_data_padded=pad_sequences(vectorized_data, padding='post', maxlen=max(lengths))\n",
    "print(\"New shape:\", vectorized_data_padded.shape)\n",
    "print(\"First example:\", vectorized_data_padded[0])\n",
    "\n",
    "# Even with the sparse output format, the shape has to be similar to the one-hot encoding\n",
    "vectorized_labels_padded=numpy.expand_dims(pad_sequences(vectorized_labels, padding='post', maxlen=max(lengths)), -1)\n",
    "print(\"Padded labels shape:\", vectorized_labels_padded.shape)\n",
    "pprint(label_map)\n",
    "print(\"First example labels:\")\n",
    "pprint(vectorized_labels_padded[0])\n",
    "\n",
    "weights = numpy.copy(vectorized_data_padded)\n",
    "weights[weights > 0] = 1\n",
    "print(\"First weight vector:\", weights[0])\n",
    "\n",
    "# Same stuff for the validation data\n",
    "validation_vectorized_data_padded=pad_sequences(validation_vectorized_data, padding='post', maxlen=max(lengths))\n",
    "validation_vectorized_labels_padded=numpy.expand_dims(pad_sequences(validation_vectorized_labels, padding='post',maxlen=max(lengths)), -1)\n",
    "validation_weights = numpy.copy(validation_vectorized_data_padded)\n",
    "validation_weights[validation_weights > 0] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Evaluation function</h4>\n",
    "<br>\n",
    "The entities can continue beyond one token so the model has to evaluate predictions based on that. We write a function that converts the sequence of tokens into entities. We then write another function that calculates precision, recall, and f-score based on the predictions. We put these functions into a custom callback which we use during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow import keras\n",
    "\n",
    "\n",
    "def _convert_to_entities(input_sequence):\n",
    "    \"\"\"\n",
    "    Reads a sequence of tags and converts them into a set of entities.\n",
    "    \"\"\"\n",
    "    entities = []\n",
    "    current_entity = []\n",
    "    previous_tag = label_map['O']\n",
    "    for i, tag in enumerate(input_sequence):\n",
    "        if tag != previous_tag and tag != label_map['O']: # New entity starts\n",
    "            if len(current_entity) > 0:\n",
    "                entities.append(current_entity)\n",
    "                current_entity = []\n",
    "            current_entity.append((tag, i))\n",
    "        elif tag == label_map['O']: # Entity has ended\n",
    "            if len(current_entity) > 0:\n",
    "                entities.append(current_entity)\n",
    "                current_entity = []\n",
    "        elif tag == previous_tag: # Current entity continues\n",
    "            current_entity.append((tag, i))\n",
    "        previous_tag = tag\n",
    "    \n",
    "    # Add the last entity to our entity list if the sentences ends with an entity\n",
    "    if len(current_entity) > 0:\n",
    "        entities.append(current_entity)\n",
    "    \n",
    "    entity_offsets = set()\n",
    "    \n",
    "    for e in entities:\n",
    "        entity_offsets.add((e[0][0], e[0][1], e[-1][1]+1))\n",
    "    return entity_offsets\n",
    "\n",
    "\n",
    "def _entity_level_PRF(predictions, gold, lengths):\n",
    "    pred_entities = [_convert_to_entities(labels[:lengths[i]]) for i, labels in enumerate(predictions)]\n",
    "    gold_entities = [_convert_to_entities(labels[:lengths[i], 0]) for i, labels in enumerate(gold)]\n",
    "    \n",
    "    tp = sum([len(pe.intersection(gold_entities[i])) for i, pe in enumerate(pred_entities)])\n",
    "    pred_count = sum([len(e) for e in pred_entities])\n",
    "    \n",
    "    try:\n",
    "        precision = tp / pred_count # tp / (tp+np)\n",
    "        recall = tp / sum([len(e) for e in gold_entities])\n",
    "        fscore = 2 * precision * recall / (precision + recall)\n",
    "    except Exception as e:\n",
    "        precision, recall, fscore = 0.0, 0.0, 0.0\n",
    "    print('\\nPrecision/Recall/F-score: %s / %s / %s' % (precision, recall, fscore))\n",
    "    return precision, recall, fscore             \n",
    "\n",
    "\n",
    "def evaluate(predictions, gold, lengths):\n",
    "    precision, recall, fscore = _entity_level_PRF(predictions, gold, lengths)\n",
    "    return precision, recall, fscore\n",
    "\n",
    "\n",
    "class EvaluateEntities(keras.callbacks.Callback):\n",
    "    def on_train_begin(self, logs={}):\n",
    "        self.losses = []\n",
    "        self.precision = []\n",
    "        self.recall = []\n",
    "        self.fscore = []\n",
    "    def on_epoch_end(self, epoch, logs={}):\n",
    "        self.losses.append(logs.get('loss'))\n",
    "        pred = numpy.argmax(self.model.predict(validation_vectorized_data_padded), axis=-1)\n",
    "        evaluation_parameters=evaluate(pred, validation_vectorized_labels_padded, validation_lengths)\n",
    "        self.precision.append(evaluation_parameters[0])\n",
    "        self.recall.append(evaluation_parameters[1])\n",
    "        self.fscore.append(evaluation_parameters[2])\n",
    "        return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Predicting word labels with word embeddings</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we get into RNNs, we first build a simpler model without any recurrent layers:\n",
    "<br>\n",
    "- input: sequence of `sequence_length` integers corresponding to words<br>\n",
    "- embedding: pretrained mapping from integers to `vector_size`-dimensional vectors<br>\n",
    "- hidden: `hidden_size`-dimensional fully connected layer with relu activation<br>\n",
    "- output: `class_count`-dimensional fully connected layer with softmax activation<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/smp/Library/Python/3.7/lib/python/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.layers import Input, Dense, Embedding, Activation, TimeDistributed\n",
    "from tensorflow.keras.optimizers import SGD, Adam\n",
    "\n",
    "\n",
    "example_count, sequence_len = vectorized_data_padded.shape\n",
    "class_count = len(label_set)\n",
    "hidden_size = 100\n",
    "\n",
    "vector_size= pretrained.shape[1]\n",
    "\n",
    "\n",
    "def build_model(example_count, sequence_len, class_count, hidden_size, vocabulary, vector_size, pretrained):\n",
    "    inp=Input(shape=(sequence_len,))\n",
    "    embeddings=Embedding(len(vocabulary), vector_size, mask_zero=True, trainable=False, weights=[pretrained])(inp)\n",
    "    hidden = TimeDistributed(Dense(hidden_size, activation=\"relu\"))(embeddings) # We change this activation function\n",
    "    outp = TimeDistributed(Dense(class_count, activation=\"softmax\"))(hidden)\n",
    "    return Model(inputs=[inp], outputs=[outp])\n",
    "\n",
    "\n",
    "model = build_model(example_count, sequence_len, class_count, hidden_size, vocabulary, vector_size, pretrained)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_2 (InputLayer)         (None, 113)               0         \n",
      "_________________________________________________________________\n",
      "embedding_2 (Embedding)      (None, 113, 300)          15000600  \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (None, 113, 100)          160400    \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 113, 4)            404       \n",
      "=================================================================\n",
      "Total params: 15,161,404\n",
      "Trainable params: 160,804\n",
      "Non-trainable params: 15,000,600\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Train the model</h3>\n",
    "<br>\n",
    "We'll then compile and train our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/smp/Library/Python/3.7/lib/python/site-packages/tensorflow_core/python/ops/math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
      "WARNING:tensorflow:From /Users/smp/Library/Python/3.7/lib/python/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
      "\n",
      "Epoch 1/10\n",
      " - 7s - loss: 0.0696\n",
      "\n",
      "Precision/Recall/F-score: 0.5664425232148576 / 0.35239043824701194 / 0.43448360555077975\n",
      "Epoch 2/10\n",
      " - 7s - loss: 0.0294\n",
      "\n",
      "Precision/Recall/F-score: 0.5628946090335114 / 0.4617529880478088 / 0.507332020135697\n",
      "Epoch 3/10\n",
      " - 8s - loss: 0.0257\n",
      "\n",
      "Precision/Recall/F-score: 0.5690555291884195 / 0.47768924302788845 / 0.5193848819579814\n",
      "Epoch 4/10\n",
      " - 7s - loss: 0.0247\n",
      "\n",
      "Precision/Recall/F-score: 0.575212866603595 / 0.48446215139442234 / 0.5259515570934256\n",
      "Epoch 5/10\n",
      " - 7s - loss: 0.0241\n",
      "\n",
      "Precision/Recall/F-score: 0.5727568572756857 / 0.49083665338645416 / 0.528641922334263\n",
      "Epoch 6/10\n",
      " - 7s - loss: 0.0238\n",
      "\n",
      "Precision/Recall/F-score: 0.5852567121997174 / 0.4950199203187251 / 0.5363695229872653\n",
      "Epoch 7/10\n",
      " - 7s - loss: 0.0235\n",
      "\n",
      "Precision/Recall/F-score: 0.5926100258884444 / 0.501593625498008 / 0.5433164311144676\n",
      "Epoch 8/10\n",
      " - 7s - loss: 0.0233\n",
      "\n",
      "Precision/Recall/F-score: 0.5948356807511737 / 0.5047808764940239 / 0.5461206896551724\n",
      "Epoch 9/10\n",
      " - 9s - loss: 0.0231\n",
      "\n",
      "Precision/Recall/F-score: 0.5965735742783385 / 0.5063745019920318 / 0.5477857989440793\n",
      "Epoch 10/10\n",
      " - 8s - loss: 0.0230\n",
      "\n",
      "Precision/Recall/F-score: 0.6024464831804281 / 0.5101593625498008 / 0.5524754611153059\n"
     ]
    }
   ],
   "source": [
    "optimizer = Adam(lr=0.001) # define the learning rate\n",
    "model.compile(optimizer=optimizer, loss=\"sparse_categorical_crossentropy\", sample_weight_mode='temporal')\n",
    "evaluation_function=EvaluateEntities()\n",
    "\n",
    "# train\n",
    "vanilla_hist=model.fit(\n",
    "    vectorized_data_padded,\n",
    "    vectorized_labels_padded,\n",
    "    sample_weight=weights,\n",
    "    batch_size=100,\n",
    "    verbose=2,\n",
    "    epochs=10,\n",
    "    callbacks=[evaluation_function]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the f-scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No handles with labels found to put in legend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "History: [0.43448360555077975, 0.507332020135697, 0.5193848819579814, 0.5259515570934256, 0.528641922334263, 0.5363695229872653, 0.5433164311144676, 0.5461206896551724, 0.5477857989440793, 0.5524754611153059]\n",
      "Highest f-score: 0.5524754611153059\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3de3BV53nv8e+j+wXdJUAICYEN2BiMARk7vkDqxBeSGJo6SZ2kTZzpGWdOj+s0kzTHaebk4jSd3E560hNPZzxu2pwmE8dJ0xZisOOenFg4sR2Ebe5GAiyQhICtK7qgy9Z+zh/aAqHIsAGJtbX37zOjYa93ra39aFn8WH7Xu97X3B0REUlcKUEXICIi00tBLyKS4BT0IiIJTkEvIpLgFPQiIgkuLegCJiotLfXq6uqgyxARmVF27tzZ5u5lk+2Lu6Cvrq6mrq4u6DJERGYUMzv6dvvUdSMikuAU9CIiCU5BLyKS4BT0IiIJTkEvIpLgFPQiIglOQS8ikuDibhy9iEgyiUSchlO97GjswAw+esuCKf8MBb2IyFU0MDzCG01d7DzaSV1jBzuPdnJ6IAzAqqpCBb2IyEzT1jtIXWMnO492sKOxk33HuxkeGV3wafHsWbz3xnJqFhRTU11EVXHOtNSgoBcRmSLuzuFQHzuPdlDX2End0U7eausDICM1hZWVBfzZHYu4ubqI1VVFFOVmXJW6FPQiIpdpMDzC3pZu6ho72dHYyWvHOunoGwKgKCedNQuK+eObK7m5uojlFQVkpqUGUqeCXkQkRl39Q6N969H+9V3N3QyFIwBUl+Rw13WzqVlQRE11MdeU5WJmAVc8SkEvIjIJd6ep4ww7GjvOBnvDqV4A0lKM5RUFfOzWBdRUF7FmQTFleZkBV/z2FPQiIsDwSIT9x0+fDfW6o52EegYByMtKY82CIjbdNI+a6mJWzi8kOyOYbpjLoaAXkYQ3FI7Q0TdEe98gHX1Do697o3/2DdHY1scbTV2cGR4BoKIwm9uvKWFNdTE3VxexZHYeKSnx0Q1zORT0IjLjDAyPnAvsviE6+gZp742+7j3XNra/JzpOfaIUg+LcDMoLsvnjmytZs6CImuoiyguyr/JPNL0U9CISuDNDI2evttvHBfVYcI8FdnvfIB29Q/QNjUz6fdJSjKLcDEpyMyjOzWDF/MKzr4vHtZfMyqAkN5OC7PQZfaUeKwW9iEy7noFhjrb3c6yjn8b2Po61j/7Z3HmG9t6hs10mE2WkppwL6VkZLCjJGRfYmdHAHgvxTPKz0+JmpEs8UdCLyBVzdzr7h88L8aPt/RyN/tkeHVs+pnRWBgtKcqlZUETprEyKZ50L77NBPiuDvEwF91RQ0ItITCIR52TPwHkBfrS9n6MdfRxt66dn8Fw/uBmU52exoCSXu5fNYUFJLgtKcqJfuczKVPRcTTrbInJWeCRCS9eZ88K8sb2fYx2jrwejDwfBaH/4/KJsFpTksrqqaDTMi3OoLs1hflEOWekzZ/hholPQiyQZd6exvZ/Dp3pHu1o6omEe7TMPR/zssVnpKVQVj16Fr19SRlVJLtUlOSwozmVeYRZpqVrSYiZQ0IskicOhXn6xq5XNu1o4HOo7256XlUZ1SS43VBTw3hvLWVCce7aLZXZeZlKMSkl0CnqRBNbc2c8vdreyZddx9h0/jRmsrS7moduqWV5RwIKSXIpy0nXDM8Ep6EUSzKnTAzy7ZzTcXzvWBcBNlYX8j/ct470ryplbkBVwhXK1xRT0ZnYf8F0gFXjK3b8+Yf9DwLeAlmjT99z9qXH784H9wL+7+yNTULeIjNPZN8Rz+06wZddxXjnSTsTh+vJ8PnffUt63Yh5VJdOzoIXMDBcNejNLBZ4A7gaagR1mttnd90849CcXCPGvArVXVKmInKdnYJgX9p9ky67jbG9oIxxxFpXm8shdi9m4spxrZ+cFXaLEiViu6NcCh9z9CICZPQ1sYvQK/aLMbA0wB3gOqLnMOkWE0akCfvXmKbbsOs6vDp5iKByhojCbP7tzIfffOI8b5uWrv11+TyxBXwE0jdtuBm6Z5LgHzGwdUA982t2bzCwF+J/AnwDvfrsPMLOHgYcBqqqqYixdJDkMhSPU1ofYsvs4L+w/Sf/QCGV5mXxkbRX3r5zH6qpChbtc0FTdjN0C/NjdB83sk8APgLuAPwe2unvzhX4R3f1J4EmAmpoaf9sDRZJEeCTCK0c62LLrONv2tnJ6IExhTjqbbqrg/pXl3LKwhFQNe5QYxRL0LUDluO35nLvpCoC7t4/bfAr4ZvT1O4A7zezPgVlAhpn1uvtjl1+ySGKKRJydxzrZsus4W/e00tY7xKzMNO5ZNof7V87jjsWlpOsBJbkMsQT9DmCxmS1kNOAfBD4y/gAzK3f31ujmRuAAgLt/dNwxDwE1CnmRc9ydPS3dbNl1nF/sbqW1e4DMtBTeff0c7l9ZzjuXztZUAnLFLhr07h42s0eA5xkdXvl9d99nZo8Dde6+GXjUzDYCYaADeGgaaxaZ8Q6e6GHLruNs2X2co+39pKca65eU8diG63jX9XM06ZdMKXOPry7xmpoar6urC7oMkSnX2n2Gn7/WwuY3jnPwZA8pBrdfW8r9N87j3hvmUpCTHnSJMoOZ2U53n3Rkoy4bRKbRYHiE/9x/imfqmqhtCOEONQuKeHzTDWxYXk5ZXmbQJUoSUNCLTIP9x0/zTF0T//5GC139w8wryOIv/uBaPrCmUk+pylWnoBeZIl39Q2zedZxn6prY23KajNQU7rlhDh+qqeT2a0s1HFICo6AXuQKRiPObw208U9fM8/tOMBSOcMO8fL6y8QY23TSPwpyMoEsUUdCLXI6mjn5+urOZf93ZTEvXGQqy0/nI2io+sGY+yysKgi5P5DwKepEYDQyP8NzeEzxT18RvD7djBncuLuPz77mOd18/R+PdJW4p6EUuwN3Z3dzNM3VNbN51nJ6BMFXFOXzm7iU8sGY+8wqzgy5R5KIU9CKTaO8d5N9eb+Gndc0cPNlDVnoK71lezgdrKrllYbGW15MZRUEvEhUeiVDbEOKZHc3854GThCPOTZWF/O37V/C+leXkZ+mBJpmZFPSS9I6Ees/eWD3VM0hJbgafuL2aD9ZUsmSOFu+QmU9BL0mpbzDMs3ta+WldEzsaO0lNMf5gaRkfrKnkrutma5ZISSgKekka7s7Oo508U9fEL3a30j80wqKyXB7bcB1/tKqC2flaNFsSk4JeElJ4JEJr9wDHOvo51tFPY3sfL+w7yZG2PnIzUrn/xnl86Ob5rK4q0upMkvAU9DJjdZ8Zpika5GNfY9stnWcIR87NzJqeaqyqLOK/vvMa3rOinFxNAyxJRL/tErcmXpVPDPOu/uHzji/OzaCyOIcb5xfyvhvLWVCcS2VxDlUlOczNz9JcM5K0FPQSqEu9Kp9flBMN84JzQV6cQ2VxNnka/igyKQW9TKtIxGnpOjPpVfnR9n66z+iqXGS6KehlyoVHIrz6Vgfb9rby/L6ThHoGz+7TVbnI1aeglykxFI7wm8NtPLfnBL/cf4LO/mGy01N559Iy1i0po7okV1flIgFR0MtlGxgeobY+xHN7T/DCgZP0DISZlZnGu66fzYblc1m/ZDbZGZrRUSRoCnq5JH2DYX59MMS2va386s1T9A+NUJCdzr03zGXD8rncsbiUzDSFu0g8UdDLRZ0eGOZXB06xdU8rL9aHGAxHKMnNYNNNFbxnxVxuXVSiKQNE4piCXibV2TfECwdOsm1PK7851M7QSIQ5+Zk8eHMlG1aUc3N1sfraRWYIBb2cFeoZ5Jf7T7BtzwlePtLOSMSpKMzm47ct4L7l5ayqLNQ87CIzUExBb2b3Ad8FUoGn3P3rE/Y/BHwLaIk2fc/dnzKzm4B/APKBEeBr7v6TKapdpkBr9xme23uCbXtPsKOxA3dYWJrLJ9ctYsPycpZX5GsuGJEZ7qJBb2apwBPA3UAzsMPMNrv7/gmH/sTdH5nQ1g98zN0bzGwesNPMnnf3rqkoXi5PU0c/2/a2sm3vCV4/NvqfYumcPB69azEbVsxl6Zw8hbtIAonlin4tcMjdjwCY2dPAJmBi0P8ed68f9/q4mZ0CygAF/VV2JNTLtr0n2La3lb0tpwFYXpHPX927lPuWz+WaslkBVygi0yWWoK8AmsZtNwO3THLcA2a2DqgHPu3u49+Dma0FMoDDl1mrXKKDJ3rYuqeV5/ae4ODJHgBWVRXy1++5jg3Ly6kszgm4QhG5GqbqZuwW4MfuPmhmnwR+ANw1ttPMyoF/AT7u7pGJbzazh4GHAaqqqqaopOR1JNTLV7bs58X6EGZwc3UxX75/Gfcun0t5QXbQ5YnIVRZL0LcAleO253PupisA7t4+bvMp4JtjG2aWDzwLfMHdX5nsA9z9SeBJgJqaGp/sGLm4/qEw3/vVIZ7a/haZaSl8fsN1vH91BbPztHKSSDKLJeh3AIvNbCGjAf8g8JHxB5hZubu3Rjc3Agei7RnAvwH/x91/NmVVy3ncnW17T/A3v9jP8e4B/mh1BY9tuE4BLyJADEHv7mEzewR4ntHhld93931m9jhQ5+6bgUfNbCMQBjqAh6Jv/xCwDiiJDsEEeMjd35jaHyN5HTrVy5c37+OlQ21cX57P3394FTXVxUGXJSJxxNzjq6ekpqbG6+rqgi4j7vUOhvnf/7eBf3zpLbIzUvnsPUv56C1VpGkqApGkZGY73b1msn16MnaGcXe27G7la8/u5+TpQT5UM5/P3XcdpbMygy5NROKUgn4GqT/Zw5f+Yx8vH2lneUU+//Ana1hdVRR0WSIS5xT0M0DPwDDf/c8G/vm3jeRmpvE3f7icD6+t0qRiIhITBX0cc3f+/Y0W/nbrm7T1DvLgzZX81b3XUZybEXRpIjKDKOjj1IHW03zpP/bxu8YOVs4v4KmP1bCysjDoskRkBlLQx5nuM8P83Qv1/MsrR8nPSuPrf7SCD9VUanpgEblsCvo4EYk4P3+9ha9vO0B73xAfvaWKz96zlMIcddOIyJVR0MeBvS3dfGnzPnYe7WRVVSH//Im1LK8oCLosEUkQCvoAdfcP8+1fHuRHrx6lKCeDb37gRj6wer66aURkSinoAxCJOD/d2cQ3njtIV/8QH3tHNZ++ewkF2elBlyYiCUhBf5Xtbu7ii/+xjzeauri5uoivbLyFZfPygy5LRBKYgv4q6ewb4lu/PMiPf3eMktxMvvOhlbx/VYWW7BORaaegn2YjEefpHcf41vMH6RkI84nbFvKXdy8mP0vdNCJydSjop9Hrxzr50uZ97G7uZu3CYr66aTlL5+YFXZaIJBkF/TRo7x3km88d5Cd1TczOy+S7D97ExpXz1E0jIoFQ0E+xY+393P+9l+gbDPPwukU8+q7FzMrUaRaR4CiBptiW3cfpPjPMs4/ewQ3z9NCTiARPyxFNsdr6ENeX5yvkRSRuKOinUO9gmJ1HO1m3pDToUkREzlLQT6GXD7cTjjjrF5cFXYqIyFkK+im0vSFEdnoqa6q1vJ+IxA8F/RSqrQ9x66JiMtNSgy5FROQsBf0UOdbeT2N7P+uWqNtGROKLgn6KvNgQAlDQi0jcUdBPkdr6EBWF2SwqzQ26FBGR88QU9GZ2n5kdNLNDZvbYJPsfMrOQmb0R/fov4/Z93Mwaol8fn8ri48XwSISXD7ezbkmZpjkQkbhz0SdjzSwVeAK4G2gGdpjZZnffP+HQn7j7IxPeWwx8CagBHNgZfW/nlFQfJ1472knvYJj1Gj8vInEoliv6tcAhdz/i7kPA08CmGL//vcAL7t4RDfcXgPsur9T4VdsQIjXFuO1aBb2IxJ9Ygr4CaBq33Rxtm+gBM9ttZj8zs8pLea+ZPWxmdWZWFwqFYiw9ftTWt7GqslBzzItIXJqqm7FbgGp3v5HRq/YfXMqb3f1Jd69x95qyspk1aqW9d5C9x7s12kZE4lYsQd8CVI7bnh9tO8vd2919MLr5FLAm1vfOdC8dasNdwypFJH7FEvQ7gMVmttDMMoAHgc3jDzCz8nGbG4ED0dfPA/eYWZGZFQH3RNsSxov1IQpz0llRodkqRSQ+XXTUjbuHzewRRgM6Ffi+u+8zs8eBOnffDDxqZhuBMNABPBR9b4eZfZXRfywAHnf3jmn4OQLh7mxvaOOOa0tJTdGwShGJTzEtPOLuW4GtE9q+OO7154HPv817vw98/wpqjFsHWnsI9Qyq20ZE4pqejL0C28emPdC0xCISxxT0V6C2IcTSOXnMLcgKuhQRkbeloL9M/UNhdrzVyZ2L9ZCUiMQ3Bf1levVIB0MjEfXPi0jcU9BfphfrQ2SmpbB2YXHQpYiIXJCC/jLVNoS4ZVEJWelaTUpE4puC/jI0d/ZzJNTHOvXPi8gMoKC/DLX1bQCsV/+8iMwACvrLUFsforwgi2tnzwq6FBGRi1LQX6LwSITfHG5j3WKtJiUiM4OC/hK90dRFz0BYwypFZMZQ0F+i2voQKQZ3aDUpEZkhFPSX6MWGNlZWFlKQo9WkRGRmUNBfgs6+IXY3d2kSMxGZURT0l0CrSYnITKSgvwTbG0LkZ6Wxcr5WkxKRmUNBHyN3p7a+jTsWl5KWqtMmIjOHEitGDad6OXF6gDvVPy8iM4yCPka19dHVpNQ/LyIzjII+Ri/Wh7imLJeKwuygSxERuSQK+hgMDI/wu7c6dDUvIjOSgj4Gr77VwWBYq0mJyMykoI9BbX2IjLQUbl1YEnQpIiKXTEEfg9r6EGuri8nO0GpSIjLzKOgv4njXGRpO9bJuiSYxE5GZKaagN7P7zOygmR0ys8cucNwDZuZmVhPdTjezH5jZHjM7YGafn6rCr5btDRpWKSIz20WD3sxSgSeADcAy4MNmtmyS4/KATwGvjmv+IJDp7iuANcAnzaz6ysu+emrr25iTn8nSOXlBlyIiclliuaJfCxxy9yPuPgQ8DWya5LivAt8ABsa1OZBrZmlANjAEnL6ykq+ekYjz0qE27tRqUiIyg8US9BVA07jt5mjbWWa2Gqh092cnvPdnQB/QChwDvu3uHRM/wMweNrM6M6sLhUKXUv+02tXcRfeZYXXbiMiMdsU3Y80sBfgO8JlJdq8FRoB5wELgM2a2aOJB7v6ku9e4e01ZWfyEam19CDO4U6tJicgMlhbDMS1A5bjt+dG2MXnAcuDX0e6NucBmM9sIfAR4zt2HgVNm9hugBjgyBbVPu+0NbdxYUUBRbkbQpYiIXLZYruh3AIvNbKGZZQAPApvHdrp7t7uXunu1u1cDrwAb3b2O0e6auwDMLBe4FXhzin+GadF9Zpg3mrrUbSMiM95Fg97dw8AjwPPAAeAZd99nZo9Hr9ov5AlglpntY/QfjH9y991XWvTV8NtDbYxEXEEvIjNeLF03uPtWYOuEti++zbHvHPe6l9EhljNObUOIWZlp3FRZGHQpIiJXRE/GTmJsNanbrikhXatJicgMpxSbxOFQHy1dZ9RtIyIJQUE/ibHVpNYr6EUkASjoJ1HbEGJhaS6VxTlBlyIicsUU9BMMDI/wypF21i3WQ1IikhgU9BPUNXYyMKzVpEQkcSjoJ6htCJGeaty6SKtJiUhiUNBPUFsfomZBMbmZMT1iICIS9xT045w8PcCbJ3rUbSMiCUVBP87YsEotGygiiURBP872hjZKZ2Vy/dz8oEsREZkyCvqoSHQ1qXWLS0lJ0WpSIpI4FPRRe49309E3pP55EUk4Cvqosf75O/SglIgkGAV9VG19G8sr8imdlRl0KSIiU0pBD/QMDPPasU7uXKxuGxFJPAp64LeH2wlHnHUKehFJQAp6RvvnczNSWbOgKOhSRESmXNIHvbtT2xDiHdeUkJGW9KdDRBJQ0idbY3s/TR1aTUpEElfSB/3ZaQ/UPy8iCUpBXx+iqjiH6tLcoEsREZkWSR30Q+EILx9p1yRmIpLQkjro64520D80om4bEUloMQW9md1nZgfN7JCZPXaB4x4wMzezmnFtN5rZy2a2z8z2mFnWVBQ+FWrr20hLMd5xjVaTEpHEddFllMwsFXgCuBtoBnaY2WZ33z/huDzgU8Cr49rSgB8Cf+ruu8ysBBiewvqvyPaGEKsXFJGXlR50KSIi0yaWK/q1wCF3P+LuQ8DTwKZJjvsq8A1gYFzbPcBud98F4O7t7j5yhTVPiVDPIPuOn2a9hlWKSIKLJegrgKZx283RtrPMbDVQ6e7PTnjvEsDN7Hkze83MPjfZB5jZw2ZWZ2Z1oVDoEsq/fC8d0rBKEUkOV3wz1sxSgO8An5lkdxpwB/DR6J/vN7N3TTzI3Z909xp3rykruzrBW1vfRkluBjfM02pSIpLYYgn6FqBy3Pb8aNuYPGA58GszawRuBTZHb8g2A7Xu3ubu/cBWYPVUFH4lIhFne0OIO7SalIgkgViCfgew2MwWmlkG8CCweWynu3e7e6m7V7t7NfAKsNHd64DngRVmlhO9Mbse2P/7H3F17W89TVvvkKYlFpGkcNGgd/cw8AijoX0AeMbd95nZ42a28SLv7WS0W2cH8Abw2iT9+FddbcNY/7welBKRxHfR4ZUA7r6V0W6X8W1ffJtj3zlh+4eMDrGMG7X1Ia6bm8fs/LgZ0i8iMm2S7snYvsEwO492aliliCSNpAv6lw+3MzzimpZYRJJG0gV9bUOI7PRUaqq1mpSIJIfkC/r6ELcuKiYzLTXoUkREroqkCvpj7f00tver20ZEkkpSBf2LY8MqFfQikkSSKuhr60NUFGazSKtJiUgSSZqgHx6J8PLhdtYtKcNM0x6ISPJImqB//VgXvYNh1mvZQBFJMkkT9LX1IVJTjNuuVdCLSHJJnqBvCLGqspB8rSYlIkkmKYK+o2+IPS3dGm0jIkkpKYJ+e0MIdw2rFJHklBRBX1vfRmFOOisqCoIuRUTkqkv4oHcfXU3q9mtLSdVqUiKShBI+6N880cOpnkHWazUpEUlSCR/0tfWj0x7cqfHzIpKkEj/oG0IsmTOL8oLsoEsREQlEQgd9/1CYHW91sk7dNiKSxBI66F890sHQSETDKkUkqSV00L9YHyIzLYW1C4uDLkVEJDAJHfTbG0LcsqiErHStJiUiySthg76l6wyHQ32sW6zRNiKS3BI26MeGVa5X/7yIJLmEDvrygiyunT0r6FJERAIVU9Cb2X1mdtDMDpnZYxc47gEzczOrmdBeZWa9ZvbZKy04FuGRCC8damPdYq0mJSJy0aA3s1TgCWADsAz4sJktm+S4POBTwKuTfJvvANuurNTY7WruomcgrGGVIiLEdkW/Fjjk7kfcfQh4Gtg0yXFfBb4BDIxvNLM/BN4C9l1hrTF7sb6NFIM7tJqUiEhMQV8BNI3bbo62nWVmq4FKd392Qvss4L8DX7nQB5jZw2ZWZ2Z1oVAopsIvpLY+xMrKQgpytJqUiMgV34w1sxRGu2Y+M8nuLwN/5+69F/oe7v6ku9e4e01Z2ZV1t3T1D7G7uYs7Ne2BiAgAaTEc0wJUjtueH20bkwcsB34dvfE5F9hsZhuBW4APmNk3gUIgYmYD7v69qSh+Mi8daiPisF6zVYqIALEF/Q5gsZktZDTgHwQ+MrbT3buBs6lqZr8GPuvudcCd49q/DPROZ8jDaLdNXlYaK+cXTufHiIjMGBftunH3MPAI8DxwAHjG3feZ2ePRq/a44e7U1rdxx7WlpKUm7CMCIiKXJJYretx9K7B1QtsX3+bYd75N+5cvsbZL1nCqlxOnBzSsUkRknIS67B2b9kBBLyJyTkIF/Yv1Ia4py6WiUKtJiYiMSZigHxge4XdvdehqXkRkgoQJ+tMDw9x7w1zuXjYn6FJEROJKTDdjZ4LZeVn8/YdXBV2GiEjcSZigF5kOw8PDNDc3MzAw8Hv7srKymD9/PunpmmpD4puCXuQCmpubycvLo7q6+rwpr92d9vZ2mpubWbhwYYAVilxcwvTRi0yHgYEBSkpKfm9dAzOjpKRk0it9kXijoBe5iLdbvEaL2shMoaAXEUlwCnoRkQSnoBe5CHe/pHaReKOgF7mArKws2tvbfy/Ux0bdZGVlBVSZSOws3q5KzCwEHL2Cb1EKtE1ROTOdzsX5Lvl8lJWVpX3ta1+rrq6uzp44vLKxsfHMF77whcZQKBSe6kKvEv1+nJMI52KBu086B0zcBf2VMrM6d68Juo54oHNxPp2P8+l8nJPo50JdNyIiCU5BLyKS4BIx6J8MuoA4onNxPp2P8+l8nJPQ5yLh+uhFROR8iXhFLyIi4yjoRUQSXMIEvZndZ2YHzeyQmT0WdD1BMrNKM/t/ZrbfzPaZ2aeCriloZpZqZq+b2S+CriVoZlZoZj8zszfN7ICZvSPomoJkZp+O/j3Za2Y/NrOEewouIYLezFKBJ4ANwDLgw2a2LNiqAhUGPuPuy4Bbgf+W5OcD4FPAgaCLiBPfBZ5z9+uAlSTxeTGzCuBRoMbdlwOpwIPBVjX1EiLogbXAIXc/4u5DwNPApoBrCoy7t7r7a9HXPYz+Ra4ItqrgmNl84L3AU0HXEjQzKwDWAf8I4O5D7t4VbFWBSwOyzSwNyAGOB1zPlEuUoK8AmsZtN5PEwTaemVUDq4BXg60kUP8L+BwQCbqQOLAQCAH/FO3KesrMcoMuKiju3gJ8GzgGtALd7v7LYKuaeokS9DIJM5sF/Cvwl+5+Ouh6gmBm7wNOufvOoGuJE2nAauAf3H0V0Ack7T0tMyti9P/+FwLzgFwz+5Ngq5p6iRL0LUDluO350bakZWbpjIb8j9z950HXE6DbgY1m1shol95dZvbDYEsKVDPQ7O5j/4f3M0aDP1m9G3jL3UPuPgz8HLgt4JqmXKIE/Q5gsZktNLMMRm+mbA64psDY6DSL/wgccPfvBF1PkNz98+4+392rGf29+JW7J9wVW6zc/QTQZGZLo03vAvYHWFLQjgG3mllO9O/Nu0jAm9NpQRcwFdw9bGaPAM8zetf8++6+L+CygnQ78KfAHjN7I9r21+6+NcCaJH78BfCj6EXREeATAdcTGHd/1cx+BrzG6Gi110nA6RA0BYKISIJLlK4bERF5Gwp6EZEEp6AXEUlwCnoRkQSnoBcRSXAKehGRBKegFxFJcP8fgvcW61N7opAAAAAASURBVDWvBh4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "def plot_history(fscores):\n",
    "    print(\"History:\", fscores)\n",
    "    print(\"Highest f-score:\", max(fscores))\n",
    "    plt.plot(fscores)\n",
    "    plt.legend(loc='lower center', borderaxespad=0.)\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "plot_history(evaluation_function.fscore)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ideas to try yourself\n",
    "\n",
    "Try playing around with the neural network. You could, for example, try\n",
    "\n",
    "- Different activation functions\n",
    "- Altering the learning rate\n",
    "- Use different optimizers\n",
    "- Adjusting the vocabulary size of the embeddings\n",
    "\n",
    "Activation functions and optimizers supported by Keras can be found here: https://keras.io/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Including the context</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then include the context by defining and building an RNN model which has the following layers:\n",
    "<br>\n",
    "- input: sequence of `sequence_length` integers corresponding to words\n",
    "- embedding: pretrained mapping from integers to `vector_size`-dimensional vectors\n",
    "- rnn: recurrent neural network, specifically LSTM, with `rnn_size`-dimensional state\n",
    "- output: `class_count`-dimensional fully connected layer with softmax activation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import LSTM\n",
    "\n",
    "\n",
    "example_count, sequence_len = vectorized_data_padded.shape\n",
    "class_count = len(label_set)\n",
    "rnn_size = 100\n",
    "\n",
    "vector_size= pretrained.shape[1]\n",
    "\n",
    "\n",
    "def build_rnn_model(example_count, sequence_len, class_count, rnn_size, vocabulary, vector_size, pretrained):\n",
    "    inp=Input(shape=(sequence_len,))\n",
    "    embeddings=Embedding(len(vocabulary), vector_size, mask_zero=True, trainable=False, weights=[pretrained])(inp)\n",
    "    rnn = LSTM(rnn_size, activation='relu', return_sequences=True)(embeddings)\n",
    "    outp=Dense(class_count, activation=\"softmax\")(rnn)\n",
    "    return Model(inputs=[inp], outputs=[outp])\n",
    "\n",
    "\n",
    "rnn_model = build_rnn_model(example_count, sequence_len, class_count, rnn_size, vocabulary, vector_size, pretrained)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_3\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_3 (InputLayer)         (None, 113)               0         \n",
      "_________________________________________________________________\n",
      "embedding_3 (Embedding)      (None, 113, 300)          15000600  \n",
      "_________________________________________________________________\n",
      "lstm_2 (LSTM)                (None, 113, 100)          160400    \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 113, 4)            404       \n",
      "=================================================================\n",
      "Total params: 15,161,404\n",
      "Trainable params: 160,804\n",
      "Non-trainable params: 15,000,600\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print(rnn_model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Train an RNN model</h3>\n",
    "<br>\n",
    "We compile and train our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      " - 81s - loss: 0.0795\n",
      "\n",
      "Precision/Recall/F-score: 0.0 / 0.0 / 0.0\n",
      "Epoch 2/10\n",
      " - 73s - loss: 0.0441\n",
      "\n",
      "Precision/Recall/F-score: 0.5872093023255814 / 0.44262948207171315 / 0.5047705588368925\n",
      "Epoch 3/10\n",
      " - 80s - loss: 0.0261\n",
      "\n",
      "Precision/Recall/F-score: 0.6366378443595415 / 0.6306772908366534 / 0.6336435504853397\n",
      "Epoch 4/10\n",
      " - 83s - loss: 0.0207\n",
      "\n",
      "Precision/Recall/F-score: 0.7183456561922366 / 0.6193227091633466 / 0.6651690201112538\n",
      "Epoch 5/10\n",
      " - 89s - loss: 0.0184\n",
      "\n",
      "Precision/Recall/F-score: 0.7392360344446898 / 0.6669322709163347 / 0.7012252591894438\n",
      "Epoch 6/10\n",
      " - 72s - loss: 0.0170\n",
      "\n",
      "Precision/Recall/F-score: 0.7291534726620224 / 0.6880478087649402 / 0.7080045095828635\n",
      "Epoch 7/10\n",
      " - 71s - loss: 0.0163\n",
      "\n",
      "Precision/Recall/F-score: 0.7565017261219793 / 0.654780876494024 / 0.7019754404698345\n",
      "Epoch 8/10\n",
      " - 71s - loss: 0.0154\n",
      "\n",
      "Precision/Recall/F-score: 0.7517543859649123 / 0.6828685258964143 / 0.7156576200417536\n",
      "Epoch 9/10\n",
      " - 72s - loss: 0.0147\n",
      "\n",
      "Precision/Recall/F-score: 0.7608360836083609 / 0.6888446215139442 / 0.7230527966544694\n",
      "Epoch 10/10\n",
      " - 69s - loss: 0.0143\n",
      "\n",
      "Precision/Recall/F-score: 0.7516240796881767 / 0.6914342629482072 / 0.7202739157501556\n"
     ]
    }
   ],
   "source": [
    "optimizer=Adam(lr=0.001) # define the learning rate\n",
    "rnn_model.compile(optimizer=optimizer,loss=\"sparse_categorical_crossentropy\", sample_weight_mode='temporal')\n",
    "\n",
    "evaluation_function=EvaluateEntities()\n",
    "\n",
    "# train\n",
    "rnn_hist=rnn_model.fit(vectorized_data_padded,vectorized_labels_padded, sample_weight=weights, batch_size=100,verbose=2,epochs=10, callbacks=[evaluation_function])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the f-scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No handles with labels found to put in legend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "History: [0.0, 0.5047705588368925, 0.6336435504853397, 0.6651690201112538, 0.7012252591894438, 0.7080045095828635, 0.7019754404698345, 0.7156576200417536, 0.7230527966544694, 0.7202739157501556]\n",
      "Highest f-score: 0.7230527966544694\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAeUklEQVR4nO3dfXRU933n8fdXM3oEIUBIGPMk8WAI6ydsASZ2G5vEu3iTtZuTNMV5tBvb7TbO82nq1Hvcs+5Jt033JG3O8dmGOM46aRLH8aYb2qXx7tZO9qSBAWHj2GBwsCSDMBgxIyQhoYeZ+e4fMwghhDXAiDtz5/M6R4e59/4082UO+ujyvb+5P3N3RESk+JUFXYCIiOSHAl1EJCQU6CIiIaFAFxEJCQW6iEhIRIN64Tlz5nhTU1NQLy8iUpR27dp13N0bJjoWWKA3NTXR2toa1MuLiBQlM3vjfMfUchERCQkFuohISCjQRURCQoEuIhISCnQRkZBQoIuIhIQCXUQkJAKbhy4ikk/JVJq+wSR9g0l6B0foHRw5s31qhJNDSdLulEfKqIiUUR4xoqcfR43ySBnRsjIqso/Ls2POPD57uyJSRnR02zCzoN8CBbqIBC+ddvqHz4RxJohH6D2V/fOs/ZmA7stun94/MJwK9O9QHjGiZZlwr4hmQv904FdEzt6+75Zm/u2/uSLvNSjQRSSvUmnn+MkhjvQMcrTnFG+eGCTRPzwazKMBfepMcPcNJZlsrZ2KaBkzqqLUVpWP/jl3RhUzqsqpPb2/OvNnbVV0dP+Z41HKzBhJpxlJOSPJ9NmPU9nHqczj4ex2cnQ7My6ZPvN4JJUmmXaGR7//7Oc43+OpOptXoItIzlJpp6tviCM9pzjSM3gmtHsGOZr9eqt3kGT67HSOlNloqNZWZoJ34eya0cCd8TaBfHq7qjySl79DZVmEyihQmZenKyg5BbqZbQT+FogAj7v7X447/nXgtuxmDdDo7jPzWahIoRlJpTk5mOTkUKYNcHIoycmhzFnn0Eia6ooI0yojTKuIMq0y+1URYVpllOryCGVlwfdcx0qm0nSdHOLNE5lgPh3aYx8f6xsiNS6sq8rLuLKumivqqli3ZPbo43l1VVxRV8WVddXMrCkviB5z2E0a6GYWAR4Dbgc6gZ1mtsXd954e4+6fHzP+08DqKahV5JK5O0PJdCZ8Jwjjk4NJ+sYdyxwfOed7hpLpS6plWkWEmjEhnwn+zL7pFVFqKiNMr4xSUxFlemWEmtFfDGePn5YdWxk9/xlsMpXmrb4hjp4+sz6RPbvuPTUa4Mf6BhmX1VSXR5g3MxPO71w6hytnVo2G9by6aubVVVFXrbAuFLmcoa8FDrh7G4CZPQXcBew9z/i7gT/LT3kiuRlOptnZkWDb63ESA8OjwTsa0NmwPjmUZCQ1+cLo0WyLYHpVlOmV5dRWRmmsrWLJnMy+2soo0ytPH8+0EqZXlo9uV0bLGBxJ0T+con8oSf9Q5qLdyaEkA8NJTg6lGBhK0j+cpH8oO2Y4yfGTw/QnBjLjh1L0DyfPCdnzKY9YNvyj1GR/Sbg7R3sH6eobOud5aioio8H8W8vnZM+oq0cDfN6MamZURxXWRSSXQJ8PHBqz3Qmsm2igmS0GmoHnLr00kbf3Vu8gP99/jOf2HeOXvzlO/3CKSJkxq6b8rLCdP7Oa2qraCQL4TCjXjgnj2qpMIBdCkLk7gyPpbPBnw3+Cx6O/LIayvyyGk/QPp3B3VlxRmwnqMWfWV9RVMaNKYR02+b4ougl4xt0nnD9kZg8ADwAsWrQozy8tYZdKO7sPnRgN8T1v9gIwr66Ku1bP57YVjdy8rJ6aivBc6zczqisiVFdEmDM9hFfxJK9y+Zd/GFg4ZntBdt9ENgGfOt8TuftmYDNAS0tLjv+RlFJ2YmCYX7zWxfP7jvGL17roHhihzODGxbP40sYVbFjZyIq5tTrTFCG3QN8JLDezZjJBvgn48PhBZrYSmAVsy2uFUlLcnVeP9PH8/mM8v+8YLxzsJu0we1oFt65o5LaVjbxreQN1NeVBlypScCYNdHdPmtmDwLNkpi0+4e57zOxRoNXdt2SHbgKecp/s4wEiZ+sfSvKvB47z/P4ufr7/GEd6BgG4ev4MHrxtGbeubOS6BTOJFNg0P5FCY0Hlb0tLi2tN0dLVcbyf5/Yd4/n9x4i1JRhOpZleGeWWZXPYsLKRd61oYO6MqqDLFCk4ZrbL3VsmOhaeq0dS0IaSKXa0J3h+XxfP7z9G+/F+AJY2TOPj6xezYWUjLU2zqYjqBqAiF0uBLlPmaM/gaC/8lweOMzCcoiJaxvol9Xxi/WI2rJzLovqaoMsUCQ0FuuRNZlphN8/tO8Zz+7p49UhmWuGVdVW8f/V8NqxsZP3ScE0rFCkk+smSi+buHEwMsO31OL96Pc7/+00XJwZGiJQZNy6exZ9sXMmGlY1cNXe6phWKXAYKdLkgh0+cygb4cba/HufN7IyUOdMr2bCykdtWNPLbmlYoEggFurytY72DbGuL86sDcba1xTmYGABgVk0565fW8x+X1LN+aT1LG3QWLhI0BbqcJX5yiO1tCX71+nG2tcVp68rMRqmtinLTknrueWcT65fWs2JubcHd/lWk1CnQS1zPwAjb2+Nsez3ztf+tPiBza9e1zbPZtGYh65fMYdWVM/TBHpECp0AvMX2DI+zsSIy2UPYe6cU9s0jBmqbZ3Hn9laxfWs818+soj2hOuEgxUaCH3MBwktaO7kwf/PU4rxzuIZV2KqJl3LBoJp9791WsX1rPdQvr3naBBBEpfAr0kBkcSfHCwW62Z6cSvtR5gpGUEy0zrl84kz+6dSnrl9Rzw+JZeVujUUQKgwK9yA0n07zUeWJ0KuELB08wnExTZnDNgpl88pYlvHNpPS1Ns/SBHpGQ0094EevqG+Lub23nwLGTmMGqeTP4+E2LeeeyetY0zaa2SnPBRUqJAr1IdfcP89HHYxzuPsXf/N713LqigZk1FUGXJSIBUqAXod7BET7+xA7a4/1855413LxsTtAliUgB0Ly0ItM/lOTe7+xk39Fe/u6jNyjMRWSUAr2IDI6kuO/JVl482M03Nq1mw8q5QZckIgVELZciMZRM8Qff28X29jhf+9B13HHNvKBLEpECozP0IpBMpfnMD1/kF6918Rfvv4b3r14QdEkiUoByCnQz22hm+83sgJk9dJ4xHzKzvWa2x8x+kN8yS1cq7Xzxxy/x7J63+LP/sIq71y4KuiQRKVCTtlzMLAI8BtwOdAI7zWyLu+8dM2Y58GXgZnfvNrPGqSq4lKTTzp/+5GV+uvtNvrRxBffe3Bx0SSJSwHI5Q18LHHD3NncfBp4C7ho35n7gMXfvBnD3Y/kts/S4O//5H/fwo9ZDfGbDMv7o1mVBlyQiBS6XQJ8PHBqz3ZndN9ZVwFVm9q9mtt3MNk70RGb2gJm1mllrV1fXxVVcAtydv/zZPp7c9gb33dLM52+/KuiSRKQI5OuiaBRYDtwK3A18y8xmjh/k7pvdvcXdWxoaGvL00uHzjX85wDd/0cZH1i3i4fe+QysBiUhOcgn0w8DCMdsLsvvG6gS2uPuIu7cDr5EJeLlA3/zF63z9/77GB25YwJ/fdbXCXERylkug7wSWm1mzmVUAm4At48b8TzJn55jZHDItmLY81lkSvrutg//yz/t437Xz+OoHr9USbyJyQSYNdHdPAg8CzwKvAk+7+x4ze9TM7swOexaIm9le4Hngj909PlVFh9HTrYd45Kd7eM875vL137tey72JyAUzdw/khVtaWry1tTWQ1y40P919mM/9aDe3LJvDtz7eooUnROS8zGyXu7dMdEyfFA3Yz145yheefok1TbPZ/DGFuYhcPAV6gH6+/xif/uELXDO/jifuWUN1hcJcRC6eAj0g216P8wff28XyxlqevHct0yt1nzQRuTQK9ADseqObTz65k0Wza/jeJ9dSV6Ol4kTk0inQL7NXDvdwzxM7aKyt5Pv3raN+emXQJYlISCjQL6P9R/v42LdjzKgu5/v330TjjKqgSxKREFGgXyZtXSf5yOMxyiNl/OD+dcyfWR10SSISMgr0y+BQYoCPPB7D3fnB/etYXD8t6JJEJIQ0tWKKHek5xYcf387AcIof3n8Tyxprgy5JREJKZ+hTqKtviI88HqO7f4Tv/v5aVl05I+iSRCTEFOhTpLt/mI99O8aRE4N85941XLfwnLsJi4jklVouU6B3cISPP7GDtuP9PPGJNaxpmh10SSJSAnSGnmf9Q0nu/c5OXj3Sy9999AZuWT4n6JJEpEQo0PNocCTF/d9t5cWD3Xzj7tVsWDk36JJEpISo5ZInQ8kUf/j3u9jWFudrH7qOf3/NvKBLEpESozP0PEim0nz2h7v5+f4uvvI71/D+1QuCLklESpAC/RKl0s4Xf/wSP9tzlEfet4oPr1sUdEkiUqIU6JcgnXYe/oeX+enuN/njf7eC37+lOeiSRKSE5RToZrbRzPab2QEze2iC4/eYWZeZ7c5+3Zf/UguLu/PoP+3lqZ2H+PSGZXzqtmVBlyQiJW7Si6JmFgEeA24HOoGdZrbF3feOG/ojd39wCmosSN/+ZTv//Vcd3HdLM1+4/aqgyxERyekMfS1wwN3b3H0YeAq4a2rLKnz/8OJhblw8i4ff+w7MLOhyRERyCvT5wKEx253ZfeN9wMx+bWbPmNnCiZ7IzB4ws1Yza+3q6rqIcgtDz6kR9h7p5beWz1GYi0jByNdF0X8Emtz9WuD/AE9ONMjdN7t7i7u3NDQ05OmlL7/WjgTusK65PuhSRERG5RLoh4GxZ9wLsvtGuXvc3Yeym48DN+anvMIUa09QESlj9SLdcEtECkcugb4TWG5mzWZWAWwCtowdYGZjPxZ5J/Bq/kosPLH2BNctrKOqPBJ0KSIioyYNdHdPAg8Cz5IJ6qfdfY+ZPWpmd2aHfcbM9pjZS8BngHumquCgnRxK8srhHrVbRKTg5HQvF3ffCmwdt++RMY+/DHw5v6UVpl1vdJNKO+uW6Ja4IlJY9EnRCxRrixMpM25YNCvoUkREzqJAv0A72hNcM7+OaZW6UaWIFBYF+gU4NZzipc4TareISEFSoF+AFw92M5Jy1jUr0EWk8CjQL8D29gRlBi1aI1RECpAC/QLsaI+z6soZzKgqD7oUEZFzKNBzNJRM8eLBE5p/LiIFS4Geo5cO9TCUTLNW/XMRKVAK9BztaI8DsFb9cxEpUAr0HMXaE6y8opZZ0yqCLkVEZEIK9ByMpNLseqNb0xVFpKAp0HPw8uEeBoZTrNUFUREpYAr0HOxoTwDogqiIFDQFeg5ibXGWNkyjobYy6FJERM5LgT6JVNpp7ehm3RK1W0SksCnQJ7H3zV76hpK6ICoiBU+BPolYdv65PiEqIoVOgT6JWHuCxfU1XFFXFXQpIiJvK6dAN7ONZrbfzA6Y2UNvM+4DZuZm1pK/EoOTTjs7OxJqt4hIUZg00M0sAjwG3AGsAu42s1UTjKsFPgvE8l1kUPa/1ceJgRHNPxeRopDLGfpa4IC7t7n7MPAUcNcE4/4c+CtgMI/1Ber0/HOdoYtIMcgl0OcDh8Zsd2b3jTKzG4CF7v6/8lhb4GLtcebPrGbh7JqgSxERmdQlXxQ1szLga8AXcxj7gJm1mllrV1fXpb70lHJ3drQn9OlQESkauQT6YWDhmO0F2X2n1QJXAz83sw7gJmDLRBdG3X2zu7e4e0tDQ8PFV30ZvN51kuMnh9VuEZGikUug7wSWm1mzmVUAm4Atpw+6e4+7z3H3JndvArYDd7p765RUfJnETvfP9QlRESkSkwa6uyeBB4FngVeBp919j5k9amZ3TnWBQYm1JWisraSpXv1zESkO0VwGuftWYOu4fY+cZ+ytl15WsNydWHuctc2zMbOgyxERyYk+KTqBg4kB3uodUrtFRIqKAn0CsbZM//wmXRAVkSKiQJ/A9vY4s6dVsKxxetCliIjkTIE+gVhbgrVN6p+LSHFRoI/T2T3A4ROnWLdE7RYRKS4K9HHO3L9FF0RFpLgo0MeJtSWYURVl5RW1QZciInJBFOjjnJ5/Xlam/rmIFBcF+hhv9Q7SER9Qu0VEipICfYwz92/RBVERKT4K9DFibXGmV0ZZNW9G0KWIiFwwBfoYsfYENy6eRTSit0VEio+SK+v4ySEOHDupdouIFC0FetZOzT8XkSKnQM+KtSeoKi/jmvl1QZciInJRFOhZ29vi3Lh4FhVRvSUiUpyUXsCJgWH2v9WndouIFDUFOrCzoxt3tCC0iBQ1BTqZ+ecV0TKuWzgz6FJERC5aToFuZhvNbL+ZHTCzhyY4/odm9rKZ7TazX5rZqvyXOnV2dCS4fuFMqsojQZciInLRJg10M4sAjwF3AKuAuycI7B+4+zXufj3wVeBrea90ivQNjvDK4R4tNyciRS+XM/S1wAF3b3P3YeAp4K6xA9y9d8zmNMDzV+LUan2jm7SjBaFFpOhFcxgzHzg0ZrsTWDd+kJl9CvgCUAFsmOiJzOwB4AGARYsWXWitUyLWliBaZqxepP65iBS3vF0UdffH3H0p8CfAfzrPmM3u3uLuLQ0NDfl66Uuyoz3OtQvqqKnI5XebiEjhyiXQDwMLx2wvyO47n6eA37mUoi6XgeEkv+7sUbtFREIhl0DfCSw3s2YzqwA2AVvGDjCz5WM23wv8Jn8lTp0X3jhBMu2afy4ioTBpn8Hdk2b2IPAsEAGecPc9ZvYo0OruW4AHzew9wAjQDXxiKovOl1h7nDKDGxfPCroUEZFLllPj2N23AlvH7XtkzOPP5rmuyyLWnuDq+XXUVpUHXYqIyCUr2U+KDo6k2H3ohNotIhIaJRvouw+dYDiZ1g25RCQ0SjbQY20JzGBNk87QRSQcSjbQd3TEWXnFDOpq1D8XkXAoyUAfTqbZ9Ua3+uciEiolGegvHz7B4EhagS4ioVKSgb69LbMg9FoFuoiESEkG+o72BMsbp1M/vTLoUkRE8qbkAj2ZStPakWDdEp2di0i4lFyg73mzl/7hFGs1/1xEQqbkAn1He6Z/rhWKRCRsSi7QY+1xmudMo3FGVdCliIjkVUkFeirt7GhPaLqiiIRSSQX6vqO99A4mNV1RREKppAL9dP9cKxSJSBiVVKDH2hIsmFXN/JnVQZciIpJ3JRPo7s6OjoRulysioVUygf6bYydJ9A/rgqiIhFZOgW5mG81sv5kdMLOHJjj+BTPba2a/NrN/MbPF+S/10sRG++cKdBEJp0kD3cwiwGPAHcAq4G4zWzVu2ItAi7tfCzwDfDXfhV6qWFucK2ZUsWh2TdCliIhMiVzO0NcCB9y9zd2HgaeAu8YOcPfn3X0gu7kdWJDfMi+NuxNrz9y/xcyCLkdEZErkEujzgUNjtjuz+87nk8A/T3TAzB4ws1Yza+3q6sq9ykvUfryfrr4hzT8XkVDL60VRM/so0AL89UTH3X2zu7e4e0tDQ0M+X/ptjc4/1wwXEQmxaA5jDgMLx2wvyO47i5m9B3gYeJe7D+WnvPyItSeYM72CpQ3Tgi5FRGTK5HKGvhNYbmbNZlYBbAK2jB1gZquBbwJ3uvux/Jd58dydWFuctc3qn4tIuE0a6O6eBB4EngVeBZ529z1m9qiZ3Zkd9tfAdODHZrbbzLac5+kuu87uU7zZM6h2i4iEXi4tF9x9K7B13L5Hxjx+T57ryhvNPxeRUhH6T4rG2uLMrCnnqsbaoEsREZlS4Q/09gRrmmZTVqb+uYiEW6gD/UjPKQ4mBnT/FhEpCaEO9NH1Q3X/cxEpAaEO9O1tCWoro7xj3oygSxERmXKhDvRYe5yWpllE1D8XkRIQ2kDv6huiratfy82JSMkIbaCfuX+LLoiKSGkIbaDH2uPUVES4en5d0KWIiFwW4Q30tgQ3Lp5FeSS0f0URkbOEMu26+4fZ/1af2i0iUlJCGeg7Ok7fv0UXREWkdIQy0GNtCSqjZVy7QP1zESkd4Qz09jirF82kMhoJuhQRkcsmdIHeOzjC3iO9uv+5iJSc0AV6a0cCd93/XERKT+gCPdaWoDxirF44K+hSREQuq/AFenuC6xbMpLpC/XMRKS05BbqZbTSz/WZ2wMwemuD4b5vZC2aWNLMP5r/M3PQPJXn5cI/aLSJSkiYNdDOLAI8BdwCrgLvNbNW4YQeBe4Af5LvAC7HrjW5SadcFUREpSbksEr0WOODubQBm9hRwF7D39AB378geS09BjTmLtceJlBk3LFb/XERKTy4tl/nAoTHbndl9BWdHe4Kr59cxvTKX31MiIuFyWS+KmtkDZtZqZq1dXV15fe7BkRQvHerhJt2/RURKVC6BfhhYOGZ7QXbfBXP3ze7e4u4tDQ0NF/MU5/XCwW6GU2ldEBWRkpVLoO8ElptZs5lVAJuALVNb1oWLtSUwgxsXK9BFpDRNGujungQeBJ4FXgWedvc9Zvaomd0JYGZrzKwT+F3gm2a2ZyqLnsiO9gSr5s2grrr8cr+0iEhByOnqobtvBbaO2/fImMc7ybRiAjGUTPHCwW4+sm5xUCWIiAQuFJ8U/XVnD0NJ9c9FpLSFItBjbXEA1jQp0EWkdIUj0NsTrJhby+xpFUGXIiISmKIP9JFUml1vdKvdIiIlr+gD/ZXDPQwMp3T/FhEpeUUf6LH2zILQa5p1/xYRKW1FH+g72hMsaZhGY21V0KWIiASqqAM9lXZ2tifUbhERocgD/dUjvfQNJVmnG3KJiBR3oJ/un2uGi4hIsQd6W5xFs2uYV1cddCkiIoEr2kBPp50dHQm1W0REsoo20F871seJgRHWKtBFRIAiDvQd2f75TUs0w0VEBIo40GNtCa6sq2LBLPXPRUSgSAPd3Ym1x1m3pB4zC7ocEZGCUJSB/npXP8dPDqt/LiIyRlEG+un+uWa4iIicUZSBHmuP01BbSfOcaUGXIiJSMHIKdDPbaGb7zeyAmT00wfFKM/tR9njMzJryXehp7k6sLTP/XP1zEZEzJg10M4sAjwF3AKuAu81s1bhhnwS63X0Z8HXgr/Jd6GkHEwMc7R1Uu0VEZJxcztDXAgfcvc3dh4GngLvGjbkLeDL7+Bng3TZFp89n7t+i+eciImPlEujzgUNjtjuz+yYc4+5JoAc4J3HN7AEzazWz1q6urosqeFZNBbevmsvyxukX9f0iImEVvZwv5u6bgc0ALS0tfjHPcfuqudy+am5e6xIRCYNcAv0wsHDM9oLsvonGdJpZFKgD4nmpUGSKjYyM0NnZyeDg4DnHqqqqWLBgAeXl5QFUJnJhcgn0ncByM2smE9ybgA+PG7MF+ASwDfgg8Jy7X9QZuMjl1tnZSW1tLU1NTWfNnHJ34vE4nZ2dNDc3B1ihSG4m7aFne+IPAs8CrwJPu/seM3vUzO7MDvs2UG9mB4AvAOdMbRQpVIODg9TXn3sbCTOjvr5+wjN3kUKUUw/d3bcCW8fte2TM40Hgd/Nbmsjlc75JWfqsgxSTovykqIiInEuBLiISEgp0ETIXQC9kv0ghUqBLyauqqiIej58T3qdnuVRVVQVUmciFsaDOQMysC3jjIr99DnA8j+UUO70fZ7ug96OhoSH6la98pampqal6/LTFjo6OUw8//HBHV1dXcioKvQz0b+NsYXg/Frt7w0QHAgv0S2Fmre7eEnQdhULvx9n0fpyh9+JsYX8/1HIREQkJBbqISEgUa6BvDrqAAqP342x6P87Qe3G2UL8fRdlDFxGRcxXrGbqIiIyjQBcRCYmiC/TJFqwuFWa20MyeN7O9ZrbHzD4bdE2FwMwiZvaimf1T0LUEzcxmmtkzZrbPzF41s/VB1xQUM/t89ufkFTP7oZmF8tNiRRXoOS5YXSqSwBfdfRVwE/CpEn4vxvosmds8C/wt8DN3XwlcR4m+L2Y2H/gM0OLuVwMRMus6hE5RBTq5LVhdEtz9iLu/kH3cR+aHdfxaryXFzBYA7wUeD7qWoJlZHfDbZNYqwN2H3f1EsFUFKgpUZ1dUqwHeDLieKVFsgZ7LgtUlx8yagNVALNhKAvc3wJeAdNCFFIBmoAv4TrYF9biZTQu6qCC4+2HgvwIHgSNAj7v/72CrmhrFFugyjplNB/4H8Dl37w26nqCY2fuAY+6+K+haCkQUuAH4b+6+GuinRFcSM7NZZP4n3wxcCUwzs48GW9XUKLZAz2XB6pJhZuVkwvz77v6ToOsJ2M3AnWbWQaYVt8HM/j7YkgLVCXS6++n/tT1DJuBL0XuAdnfvcvcR4CfAOwOuaUoUW6CPLlhtZhVkLmxsCbimQFjmtoDfBl51968FXU/Q3P3L7r7A3ZvI/Lt4zt1DeRaWC3c/ChwysxXZXe8G9gZYUpAOAjeZWU325+bdhPQCcU5rihYKd0+a2ekFqyPAE+6+J+CygnIz8DHgZTPbnd33p9n1X0UAPg18P3vy0wbcG3A9gXD3mJk9A7xAZnbYi4T0FgD66L+ISEgUW8tFRETOQ4EuIhISCnQRkZBQoIuIhIQCXUQkJBToIiIhoUAXEQmJ/w8B/auPTg2RHgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "plot_history(evaluation_function.fscore)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By including the context, our f-score increase by over 15 points! This isn't good performance, but should demonstrate the value of recurrent networks for the task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ideas to try yourself\n",
    "\n",
    "Try altering the structure of the network. You could, for example\n",
    "\n",
    "- Try different RNNs (LSTM, biLSTM)\n",
    "- Try using CNNs (hint: use the Conv1D layer, and the Dense layer afterwards needs a TimeDistributed wrapper layer)\n",
    "\n",
    "Again, the layers supported by Keras can be found from its documentation: https://keras.io/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}