bow_classifier_with_embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bag-of-words classifier with pretrained word embeddings\n",
    "\n",
    "- During the lecture we will cover the concept of embeddings and the simple word2vec method\n",
    "- If we have a trained word embeddings model, we can transfer that knowledge into a new task and model (transfer learning)\n",
    "- What we achieve here: Initialize the weights in the classifier with pretrained word embeddings\n",
    "- Word embeddings downloaded at: https://dl.fbaipublicfiles.com/fasttext/vectors-english/wiki-news-300d-1M.vec.zip\n",
    "\n",
    "### Read data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'class': 'neg', 'text': 'Just finished this movie... saw it on the video shelf and being a Nick Stahl fan I just had to rent it. In all honesty, it probably should have stayed on the shelf. The concept was an interesting one and there were several fairly smart twists and turns but somehow I guessed almost all of them before they came along. And the movie just went a little too far in the end in my opinion... if you have to suffer through a viewing of it you\\'ll see what I mean!  On a positive note, Nick Stahl\\'s acting was great (especially considering what he had to work with). Eddie Kaye Thomas was also good but he always plays the same type of character... too much Paul Finch from \\\\American Pie\\\\\" coming through for my liking.  And finally, the worst part of this movie has to be January Jones\\' emotionless performance... I guess a pretty face really is all that matters in Hollywood.\"'}\n",
      "['Just finished this movie... saw it on the video shelf and being a Nick Stahl fan I just had to rent it. In all honesty, it probably should have stayed on the shelf. The concept was an interesting one and there were several fairly smart twists and turns but somehow I guessed almost all of them before they came along. And the movie just went a little too far in the end in my opinion... if you have to suffer through a viewing of it you\\'ll see what I mean!  On a positive note, Nick Stahl\\'s acting was great (especially considering what he had to work with). Eddie Kaye Thomas was also good but he always plays the same type of character... too much Paul Finch from \\\\American Pie\\\\\" coming through for my liking.  And finally, the worst part of this movie has to be January Jones\\' emotionless performance... I guess a pretty face really is all that matters in Hollywood.\"', \"This is a excellent series. You will laugh, you will cry, these wonderful people will be a part of your family. The way this family cares for one another and helps each other through their crisis sets a great example of the way we should live our lives. There are many good things they do, and a number of bad choices, but they never turn their backs on family, they work through problems. Michelle, the youngest daughter, is the cutest thing I've seen. Stephanie, the middle daughter, suffers with Middle Child Syndrome and with the help of EVERYONE in the family it's better. DJ, the oldest daughter, is growing up whether her dad wants it to happen or not. One thing they all share is they miss their mom. Danny (Dad), Joey, and Uncle Jesse love these kids so much, and it's apparent in every episode.\"]\n",
      "['neg', 'pos']\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import random\n",
    "with open(\"data/imdb_train.json\") as f:\n",
    "    data=json.load(f)\n",
    "random.shuffle(data) \n",
    "print(data[0])\n",
    "\n",
    "# We need to gather the texts, into a list\n",
    "texts=[one_example[\"text\"] for one_example in data]\n",
    "labels=[one_example[\"class\"] for one_example in data]\n",
    "print(texts[:2])\n",
    "print(labels[:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use gensim to read the embedding model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models import KeyedVectors\n",
    "\n",
    "#Only grab the 100K most common entries\n",
    "vector_model=KeyedVectors.load_word2vec_format(\"data/wiki-news-300d-1M.vec\", binary=False, limit=100000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Words from embedding model: 100000\n",
      "First 50 words: [',', 'the', '.', 'and', 'of', 'to', 'in', 'a', '\"', ':', ')', 'that', '(', 'is', 'for', 'on', '*', 'with', 'as', 'it', 'The', 'or', 'was', \"'\", \"'s\", 'by', 'from', 'at', 'I', 'this', 'you', '/', 'are', '=', 'not', '-', 'have', '?', 'be', 'which', ';', 'all', 'his', 'has', 'one', 'their', 'about', 'but', 'an', '|']\n"
     ]
    }
   ],
   "source": [
    "# sort based on the index to make sure they are in the correct order\n",
    "words=[k for k,v in sorted(vector_model.vocab.items(), key=lambda x:x[1].index)]\n",
    "print(\"Words from embedding model:\",len(words))\n",
    "print(\"First 50 words:\",words[:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalize the vectors\n",
    "\n",
    "- Easier to learn on top of these vectors when the magnitude does not vary much"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before normalization: [-0.0234 -0.0268 -0.0838  0.0386 -0.0321  0.0628  0.0281 -0.0252  0.0269\n",
      " -0.0063]\n",
      "After normalization: [-0.0163762  -0.01875564 -0.05864638  0.02701372 -0.02246478  0.04394979\n",
      "  0.01966543 -0.0176359   0.01882563 -0.00440898]\n"
     ]
    }
   ],
   "source": [
    "print(\"Before normalization:\",vector_model.get_vector(\"in\")[:10])\n",
    "vector_model.init_sims(replace=True)\n",
    "print(\"After normalization:\",vector_model.get_vector(\"in\")[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text analyzer and vectorizer\n",
    "\n",
    "- When we use an embedding layer (keras.layers.Embedding) the input data must be a sequence, not a bag-of-words vector\n",
    "- You can use CountVectorizer only as an analyzer without building the feature matrix\n",
    "- We will then build the vectorizer part later ourselves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Just', 'finished', 'this', 'movie', 'saw', 'it', 'on', 'the', 'video', 'shelf', 'and', 'being', 'Nick', 'Stahl', 'fan', 'just', 'had', 'to', 'rent', 'it', 'In', 'all', 'honesty', 'it', 'probably', 'should', 'have', 'stayed', 'on', 'the', 'shelf', 'The', 'concept', 'was', 'an', 'interesting', 'one', 'and', 'there', 'were', 'several', 'fairly', 'smart', 'twists', 'and', 'turns', 'but', 'somehow', 'guessed', 'almost', 'all', 'of', 'them', 'before', 'they', 'came', 'along', 'And', 'the', 'movie', 'just', 'went', 'little', 'too', 'far', 'in', 'the', 'end', 'in', 'my', 'opinion', 'if', 'you', 'have', 'to', 'suffer', 'through', 'viewing', 'of', 'it', 'you', 'll', 'see', 'what', 'mean', 'On', 'positive', 'note', 'Nick', 'Stahl', 'acting', 'was', 'great', 'especially', 'considering', 'what', 'he', 'had', 'to', 'work', 'with', 'Eddie', 'Kaye', 'Thomas', 'was', 'also', 'good', 'but', 'he', 'always', 'plays', 'the', 'same', 'type', 'of', 'character', 'too', 'much', 'Paul', 'Finch', 'from', 'American', 'Pie', 'coming', 'through', 'for', 'my', 'liking', 'And', 'finally', 'the', 'worst', 'part', 'of', 'this', 'movie', 'has', 'to', 'be', 'January', 'Jones', 'emotionless', 'performance', 'guess', 'pretty', 'face', 'really', 'is', 'all', 'that', 'matters', 'in', 'Hollywood']\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "import numpy\n",
    "analyzer=CountVectorizer(lowercase=False).build_analyzer() # includes tokenizer and preprocessing\n",
    "print(analyzer(texts[0]))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Expand the vocabulary using words from the embedding model\n",
    "\n",
    "- The embedding model usually knows more words than the task specific model, because it has seen a lot more data\n",
    "- If you wish, you can use the embedding model vocabulary to expand the task specific one"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Words from embedding model: 47944\n"
     ]
    }
   ],
   "source": [
    "# init the vectorizer vocabulary using words from the embedding model\n",
    "def init_vocabulary(vocab, text, text_analyzer):\n",
    "    for word in analyzer(text):\n",
    "        vocab.setdefault(word, len(vocab))\n",
    "    return vocab\n",
    "\n",
    "words_from_model=\" \".join(words[:50000]) # use 50K words from the embedding model to initialize the vocabulary --> expands the learned vocabulary\n",
    "vocabulary={\"<SPECIAL>\": 0} # zero has a special meaning in sequence models, prevent using it for a normal word\n",
    "vocabulary=init_vocabulary(vocabulary, words_from_model, analyzer)\n",
    "print(\"Words from embedding model:\",len(vocabulary))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vectorizer\n",
    "\n",
    "- Build a dictionary to turn words into numbers, here we use the one which we initialized with the embedding model\n",
    "- Vectorizing a sequence: In our data each example is a list of words, we need to turn each example into list of numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Words in vocabulary: 105571\n",
      "Vectorized data shape: (25000,)\n",
      "First example vectorized: [23, 5824, 75, 3, 31, 2414, 74, 2185, 177, 107, 15, 29, 3, 1, 3834, 2414, 23, 463, 481, 405, 165, 109, 6573, 97, 33, 2672, 3, 12, 333, 23, 87, 5, 19132, 6, 633, 841, 12, 5, 1, 3317, 55, 12, 524, 75, 121, 725, 5317, 110, 1322, 1, 106, 32, 2225, 6, 25615, 5923, 3194, 12287, 5, 12, 113, 333, 23, 194, 2336, 84, 4, 24, 12742, 74, 47, 9328, 9, 12, 266, 47944, 47945, 47946]\n",
      "First example text: ['have', 'walked', 'out', 'of', 'about', 'movies', 'my', 'entire', 'life', 'This', 'was', 'one', 'of', 'the', 'worst', 'movies', 'have', 'ever', 'seen', 'don', 'know', 'how', 'sat', 'through', 'an', 'hour', 'of', 'it', 'must', 'have', 'been', 'in', 'coma', 'that', 'night', 'saw', 'it', 'in', 'the', 'theatre', 'when', 'it', 'came', 'out', 'years', 'ago', 'couldn', 'even', 'remember', 'the', 'name', 'but', 'knew', 'that', 'Penelope', 'Ann', 'Miller', 'starred', 'in', 'it', 'It', 'must', 'have', 'really', 'affected', 'me', 'to', 'be', 'wasting', 'my', 'time', 'commenting', 'on', 'it', 'today', 'Yech', 'Vomit', 'Barf']\n"
     ]
    }
   ],
   "source": [
    "def vectorizer(vocab, texts):\n",
    "    vectorized_data=[] # turn text into numbers based on our vocabulary mapping\n",
    "    for one_example in texts:\n",
    "        vectorized_example=[]\n",
    "        for word in analyzer(one_example):\n",
    "            vocab.setdefault(word, len(vocab)) # add word to our vocabulary if it does not exist\n",
    "            vectorized_example.append(vocab[word])\n",
    "        vectorized_data.append(vectorized_example)\n",
    "    \n",
    "    vectorized_data=numpy.array(vectorized_data) # turn python list into numpy matrix\n",
    "    return vectorized_data, vocab\n",
    "\n",
    "vectorized_data, vocabulary=vectorizer(vocabulary, texts)\n",
    "\n",
    "# now vectorized data is the same as feature_matrix, but in different format\n",
    "print(\"Words in vocabulary:\",len(vocabulary))\n",
    "print(\"Vectorized data shape:\",vectorized_data.shape)\n",
    "print(\"First example vectorized:\",vectorized_data[0])\n",
    "inversed_vocabulary={value:key for key, value in vocabulary.items()} # inverse the dictionary\n",
    "print(\"First example text:\",[inversed_vocabulary[idx] for idx in vectorized_data[0]])\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Labels into numerical vectors\n",
    "\n",
    "- Same as in the original BOW classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "class_numbers shape= (25000,)\n",
      "class_numbers [0 0 1 ... 0 0 1]\n",
      "class labels ['neg' 'pos']\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "label_encoder=LabelEncoder() #Turns class labels into integers\n",
    "class_numbers=label_encoder.fit_transform(labels)\n",
    "print(\"class_numbers shape=\",class_numbers.shape)\n",
    "print(\"class_numbers\",class_numbers)\n",
    "print(\"class labels\",label_encoder.classes_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Network\n",
    "\n",
    "- First we need to create an embedding matrix which we can then plug into the neural network\n",
    "- The embedding matrix must follow the order from the vectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found pretrained vectors for 63848 words.\n",
      "Shape of pretrained embeddings: (105571, 300)\n",
      "Vector for the word 'in': [-0.0163762  -0.01875564 -0.05864638  0.02701372 -0.02246478  0.04394979\n",
      "  0.01966543 -0.0176359   0.01882563 -0.00440898]\n"
     ]
    }
   ],
   "source": [
    "def load_pretrained_embeddings(vocab, embedding_model):\n",
    "    \"\"\" vocab: vocabulary from our data vectorizer, embedding_model: model loaded with gensim \"\"\"\n",
    "    pretrained_embeddings=numpy.random.uniform(low=-0.05, high=0.05, size=(len(vocab),embedding_model.vectors.shape[1])) # initialize new matrix (words x embedding dim)\n",
    "    found=0\n",
    "    for word,idx in vocab.items():\n",
    "        if word in embedding_model.vocab:\n",
    "            pretrained_embeddings[idx]=embedding_model.get_vector(word)\n",
    "            found+=1\n",
    "            \n",
    "    print(\"Found pretrained vectors for {found} words.\".format(found=found))\n",
    "    return pretrained_embeddings\n",
    "\n",
    "pretrained=load_pretrained_embeddings(vocabulary, vector_model)\n",
    "print(\"Shape of pretrained embeddings:\",pretrained.shape)\n",
    "print(\"Vector for the word 'in':\",pretrained[vocabulary[\"in\"]][:10])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sequential input\n",
    "\n",
    "- Remember how the shape of the input data matrix had undefined number of columns\n",
    "- Now we must make it into fixed size (same for each example)\n",
    "- Padding: include zeros until you reach the correct size\n",
    "- You will hear more about this next week!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Old shape: (25000,)\n",
      "New shape: (25000, 2366)\n",
      "First example: [  23 5824   75 ...    0    0    0]\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.sequence import pad_sequences\n",
    "print(\"Old shape:\", vectorized_data.shape)\n",
    "vectorized_data_padded=pad_sequences(vectorized_data, padding='post')\n",
    "print(\"New shape:\", vectorized_data_padded.shape)\n",
    "print(\"First example:\", vectorized_data_padded[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to define a new keras layer for later use. You should not need anything like this after next weeks when we will show you convolutional layers and LSTMs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import Layer\n",
    "from keras import backend as K\n",
    "import tensorflow as tf\n",
    "\n",
    "class MaskedAveragePooling(Layer):\n",
    "    def __init__(self, **kwargs):\n",
    "        self.supports_masking = True\n",
    "        super(MaskedAveragePooling, self).__init__(**kwargs)\n",
    "\n",
    "    def compute_mask(self, input, input_mask=None):\n",
    "        # do not pass the mask to the next layers\n",
    "        return None\n",
    "\n",
    "    def call(self, x, mask=None):\n",
    "        if mask is not None:\n",
    "            # mask (batch, time)\n",
    "            mask = K.cast(mask, K.floatx())\n",
    "            # mask (batch, x_dim, time)\n",
    "            mask = K.repeat(mask, x.shape[-1])\n",
    "            # mask (batch, time, x_dim)\n",
    "            mask = tf.transpose(mask, perm=[0,2,1])\n",
    "            x = x * mask\n",
    "        return K.sum(x, axis=1) / K.sum(mask, axis=1)\n",
    "\n",
    "    def compute_output_shape(self, input_shape):\n",
    "        # remove temporal dimension\n",
    "        return input_shape[0], input_shape[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Our network structure:\n",
    "\n",
    "- Input layer, Embedding layer with pretrained weights, Average of embeddings, Non-linear activation, Classification layer\n",
    "- The key point here is the embedding layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_3\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_5 (InputLayer)         (None, 2366)              0         \n",
      "_________________________________________________________________\n",
      "embedding_5 (Embedding)      (None, 2366, 300)         31671300  \n",
      "_________________________________________________________________\n",
      "masked_average_pooling_5 (Ma (None, 300)               0         \n",
      "_________________________________________________________________\n",
      "activation_3 (Activation)    (None, 300)               0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 2)                 602       \n",
      "=================================================================\n",
      "Total params: 31,671,902\n",
      "Trainable params: 31,671,902\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "Train on 22500 samples, validate on 2500 samples\n",
      "Epoch 1/50\n",
      "22500/22500 [==============================] - 344s 15ms/step - loss: 0.5293 - accuracy: 0.7956 - val_loss: 0.3596 - val_accuracy: 0.8680\n",
      "Epoch 2/50\n",
      "22500/22500 [==============================] - 304s 14ms/step - loss: 0.2794 - accuracy: 0.9012 - val_loss: 0.2693 - val_accuracy: 0.8988\n",
      "Epoch 3/50\n",
      "20200/22500 [=========================>....] - ETA: 30s - loss: 0.1873 - accuracy: 0.9401"
     ]
    }
   ],
   "source": [
    "from keras.models import Model\n",
    "from keras.layers import Input, Dense, Embedding, Activation\n",
    "from keras import backend as K\n",
    "from keras.layers.core import Lambda\n",
    "from keras.optimizers import SGD, Adam\n",
    "\n",
    "\n",
    "example_count,sequence_len=vectorized_data_padded.shape\n",
    "class_count=len(label_encoder.classes_)\n",
    "vector_size=pretrained.shape[1] # embedding dim (\"hidden layer\") must be the same as in the pretrained model\n",
    "\n",
    "inp=Input(shape=(sequence_len,))\n",
    "embeddings=Embedding(len(vocabulary), vector_size, mask_zero=True, weights=[pretrained])(inp)\n",
    "average_embeddings=MaskedAveragePooling()(embeddings) # custom layer we defined above\n",
    "tanh=Activation(\"tanh\")(average_embeddings)\n",
    "outp=Dense(class_count, activation=\"softmax\")(tanh)\n",
    "model=Model(inputs=[inp], outputs=[outp])\n",
    "\n",
    "optimizer=Adam(lr=0.001) # define the learning rate\n",
    "model.compile(optimizer=optimizer,loss=\"sparse_categorical_crossentropy\",metrics=['accuracy'])\n",
    "\n",
    "print(model.summary())\n",
    "\n",
    "# train\n",
    "hist=model.fit(vectorized_data_padded,class_numbers,batch_size=100,verbose=1,epochs=50,validation_split=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "History: [0.7675999927520752, 0.7840000009536743, 0.7995999956130981, 0.8043999981880188, 0.821999990940094, 0.8323999953269958, 0.842000002861023, 0.8496000003814698, 0.8607999992370605, 0.8672000002861023, 0.8744000029563904, 0.8764000058174133, 0.8792000079154968, 0.881200008392334, 0.8828000044822693, 0.8856000018119812, 0.8872000074386597, 0.8880000042915345, 0.8900000047683716, 0.8912000083923339, 0.8928000092506408, 0.8952000069618226, 0.8971999979019165, 0.8964000010490417, 0.898800003528595, 0.8980000042915344, 0.8960000038146972, 0.8984000039100647, 0.8980000019073486, 0.8984000039100647, 0.9004000043869018, 0.9000000023841858, 0.898400001525879, 0.9, 0.8984000039100647, 0.898400001525879, 0.9000000023841858, 0.9, 0.9, 0.9016000008583069, 0.9004000020027161, 0.9016000032424927, 0.9008000016212463, 0.9000000023841858, 0.903199999332428, 0.9015999984741211, 0.901199996471405, 0.8999999976158142, 0.8996000003814697, 0.9007999992370606]\n",
      "Max accuracy: 0.903199999332428\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xd4VGX2wPHvSSNAQk2oodcESCghgJTQOyKIKMoqrMrqWlZd2GXV36rsurr2si6KFMFVEEWRqiJFQKQkNCGUUAIkoaSRkED6+/vjDjHUJGSSSSbn8zw8mbnz3nvPDZMzd9773vOKMQallFIVg4ujA1BKKVV6NOkrpVQFoklfKaUqEE36SilVgWjSV0qpCkSTvlJKVSAFJn0RmSsi50Rk3w1eFxF5T0SOiMheEemc77UHRCTS9u8BewaulFKq6Apzpv8JMPQmrw8DWtn+TQFmAohILeAFoBsQArwgIjWLE6xSSqniKTDpG2M2Aok3aTIaWGAsW4EaIlIfGAKsMcYkGmOSgDXc/MNDKaVUCXOzwzYaAqfyPY+2LbvR8muIyBSsbwlUrVq1S9u2be0QllJKVRzh4eHxxhjfgtrZI+kXmzFmFjALIDg42ISFhTk4IqWUKl9E5ERh2tlj9E4M0Cjfcz/bshstV0op5SD2SPrLgPtto3i6A8nGmNPA98BgEalpu4A72LZMKaWUgxTYvSMiC4G+gI+IRGONyHEHMMZ8CKwChgNHgIvAZNtriSLyD2CHbVMzjDE3uyCslFKqhBWY9I0xEwp43QCP3eC1ucDcWwvtN1lZWURHR5Oenl7cTSl1DU9PT/z8/HB3d3d0KEqVuDJxIbcg0dHReHt707RpU0TE0eEoJ2KMISEhgejoaJo1a+bocJQqceWiDEN6ejq1a9fWhK/sTkSoXbu2fotUFUa5SPqAJnxVYvS9pSqScpP0lVJKFZ8m/ULo168f339/5WjTd955h0cfffSm63l5eQEQGxvLuHHjrtumb9++FHQz2jvvvMPFixfzng8fPpzz588XJnS7iYqK4vPPPy/VfSpVIeRkQ9IJOL4Jjm8s8d2Viwu5jjZhwgQWLVrEkCFD8pYtWrSI1157rVDrN2jQgK+++uqW9//OO+8wceJEqlSpAsCqVatueVu36nLSv/fee0t93/llZ2fj5qZvW1UOZV2CmHA4uRUSjsD5k9a/lFgwOVabBp1gyoYSDUPP9Ath3LhxrFy5kszMTMBKgLGxsfTu3ZvU1FQGDBhA586d6dChA99+++0160dFRdG+fXsALl26xD333IO/vz9jxozh0qVLee0effRRgoODadeuHS+88AIA7733HrGxsfTr149+/foB0LRpU+Lj4wF46623aN++Pe3bt+edd97J25+/vz8PP/ww7dq1Y/DgwVfs57Ivv/yS9u3bExQURJ8+fQDIyclh2rRpdO3alcDAQD766CMApk+fzqZNm+jYsSNvv/32Fdu52e9gwYIFBAYGEhQUxO9+9zsAzp49y5gxYwgKCiIoKIgtW7Zc8TsCeOONN3jxxRcB69vQU089RXBwMO+++y7Lly+nW7dudOrUiYEDB3L27Nm8OCZPnkyHDh0IDAxkyZIlzJ07l6eeeipvux9//DFPP/30Tf63lbKTS+fh8Pew5gWYMxheaQSfjIB1/7DO6o2BJrdB72fg9vfh/m/hrk9KPKxyd8r00vL9RMSm2HWbAQ2q8cKodjd8vVatWoSEhLB69WpGjx7NokWLGD9+PCKCp6cn33zzDdWqVSM+Pp7u3btz++233/Di4MyZM6lSpQoHDhxg7969dO6cN/0AL7/8MrVq1SInJ4cBAwawd+9ennzySd566y3Wr1+Pj4/PFdsKDw9n3rx5bNu2DWMM3bp1IzQ0lJo1axIZGcnChQv5+OOPGT9+PEuWLGHixIlXrD9jxgy+//57GjZsmNddNGfOHKpXr86OHTvIyMigZ8+eDB48mFdffZU33niDFStWXHNMN/odRERE8M9//pMtW7bg4+NDYqJ1b96TTz5JaGgo33zzDTk5OaSmppKUlHTT/6PMzMy8brCkpCS2bt2KiDB79mxee+013nzzTf7xj39QvXp1fv3117x27u7uvPzyy7z++uu4u7szb968vA8ypewiJ8s6cz+7H85FWD/PRkDySet1FzfrDL77o1aSb9QNqtRyWLjlLuk7yuUunstJf86cOYA1zvvZZ59l48aNuLi4EBMTw9mzZ6lXr951t7Nx40aefPJJAAIDAwkMDMx7bfHixcyaNYvs7GxOnz5NRETEFa9fbfPmzYwZM4aqVasCMHbsWDZt2sTtt99Os2bN6NixIwBdunQhKirqmvV79uzJpEmTGD9+PGPHjgXghx9+YO/evXndUcnJyURGRuLh4XHDOG70O1i3bh133XVX3odVrVrWG33dunUsWLAAAFdXV6pXr15g0r/77rvzHkdHR3P33Xdz+vRpMjMz88bX//jjjyxatCivXc2a1vQN/fv3Z8WKFfj7+5OVlUWHDh1uui+lrnHpPMQdtHXJnPita+b8STh/CnKzrHYublC7FTTqCl0egEYh0DAYPKo4Nv58yl3Sv9kZeUkaPXo0Tz/9NDt37uTixYt06dIFgM8++4y4uDjCw8Nxd3enadOmtzTm+/jx47zxxhvs2LGDmjVrMmnSpGKNHa9UqVLeY1dX1+t273z44Yds27aNlStX0qVLF8LDwzHG8P77719x/QJgw4YNN9yXPX4Hbm5u5Obm5j2/ev3LH2wATzzxBM888wy33347GzZsyOsGupGHHnqIf/3rX7Rt25bJkycXKS5VAeVkQ9wBiN4B0WHWz/jDV7apWgdqNIb6QeB/O9QJgLoB4NMa3Cpdf7tlhPbpF5KXlxf9+vXj97//PRMm/FaZIjk5mTp16uDu7s769es5ceLm1U379OmTNwpm37597N27F4CUlBSqVq1K9erVOXv2LKtXr85bx9vbmwsXLlyzrd69e7N06VIuXrxIWloa33zzDb179y70MR09epRu3boxY8YMfH19OXXqFEOGDGHmzJlkZVlnLocPHyYtLe2GMdzsd9C/f3++/PJLEhISAPK6dwYMGMDMmTMB6xpCcnIydevW5dy5cyQkJJCRkXHdbqT8+2vY0JqaYf78+XnLBw0axAcffJD3/PK3h27dunHq1Ck+//zzK/7vlAIgOwOifoYN/4b5o+DVxvBhL1jxtNUnX6s59H8e7v0SHtsBz56GaZHw8FqrD37QSxB0N9TrUOYTPpTDM31HmjBhAmPGjLmiC+G+++5j1KhRdOjQgeDgYAqaAObRRx9l8uTJ+Pv74+/vn/eNISgoiE6dOtG2bVsaNWpEz54989aZMmUKQ4cOpUGDBqxfvz5veefOnZk0aRIhISGAdUbbqVOn63blXM+0adOIjIzEGMOAAQMICgoiMDCQqKgoOnfujDEGX19fli5dSmBgIK6urgQFBTFp0qQrLobe6HfQrl07nnvuOUJDQ3F1daVTp0588sknvPvuu0yZMoU5c+bg6urKzJkz6dGjB3//+98JCQmhYcOGN/09vvjii9x1113UrFmT/v37c/z4cQCef/55HnvsMdq3b4+rqysvvPBCXrfV+PHj2b17d16Xj6rAcrLg1HaI2gRRm60z+ex0QKBee+h0H/iFgF8w1GwKTnbznlj10sqO602icuDAAfz9/R0UkXIGI0eO5Omnn2bAgAHXfV3fY07uwlk48iNE/gBH10NGMiBQPxCa9IKmvaBJD6hcfk8KRCTcGBNcUDs901dO7fz584SEhBAUFHTDhK+c1LkDsH8pRH4PsbusZV71IOB2aDUYmvWByjUcG6MDaNJXTq1GjRocPny44IbKOSQchX1fw74l1sVYBPy6Wn3yrQZDvUCn664pKk36SqnyKzsTzu6zyhfs/xpO77GWN+4Bw9+wRtZ413VsjGWMJn2lVPmQm2sNnYzdCTE7rZ9nfoUc6055GnaBwS9Duzugup9jYy3DNOkrpcqunCw48TNELIODKyH1jLXcwwvqd4Ruf4AGna0unBqNHBtrOaFJXylVtmSlw7H1cGA5HFoFl5LAvQq0HAith1h3uPq0AhdXR0daLmnSL4SEhIS8kR9nzpzB1dUVX19fALZv337TEgWXTZ48menTp9OmTZsbtvnggw+oUaMG9913n30CL6R169ZRpUoVunfvXqr7VSpPRqo1nPLAcutnZipUqg5thoL/KGgxoEyVMijPCpX0RWQo8C7gCsw2xrx61etNsCZA9wUSgYnGmGjba68BI7Du/l0D/MmUtZsDClC7dm12794NWDcGeXl5MXXq1CvaGGMwxuDicv2bnOfNm1fgfh577Lrzy5e4devW4ePj4/Ckn5OTg6urnr1VGJeS4NB3cGAZHFkLORlQxQfa32ldgG3WB9wKPqFSRVNgGQYRcQU+AIYBAcAEEQm4qtkbwAJjTCAwA3jFtu5tQE8gEGgPdAVC7Ra9gx05coSAgADuu+8+2rVrx+nTp5kyZUpeeeQZM2bkte3Vqxe7d+8mOzubGjVqMH36dIKCgujRowfnzp0DrDtKL5dH7tWrF9OnTyckJIQ2bdqwZcsWANLS0rjzzjsJCAhg3LhxBAcH530g5Tdt2jQCAgIIDAzkr3/9K2CVNB47dizBwcGEhISwdetWjh49yuzZs3n99dfp2LFj3n4u27p1Kz169KBTp0707NmTyMhIwKpr//TTT9O+fXsCAwP573//C8C2bdvo0aMHQUFBdOvWjYsXLzJ79uwryhsPHTqUzZs35/0unnrqKQIDA9m+fTsvvPACXbt2pX379jzyyCNcPj84fPgw/fv3JygoiM6dOxMVFcW99957RbmGu+++m5UrVxbvP1WVnEvn4dBq+P45+CgUXmsOSx+B03sh+PcwaRVMPQy3vwetBmrCLyGFOdMPAY4YY44BiMgiYDQQka9NAPCM7fF6YKntsQE8AQ9AAHfgbLEiXj3dumJvT/U6wLBXC253HQcPHmTBggUEB1s3wr366qvUqlWL7Oxs+vXrx7hx4wgIuPIzMjk5mdDQUF599VWeeeYZ5s6dy/Tp06/ZtjGG7du3s2zZMmbMmMF3333H+++/T7169ViyZAl79uy5ojTzZWfPnmXVqlXs378fEckrm/zkk0/yl7/8he7duxMVFcXIkSPZt28fDz30ED4+Plck5sv8/f3ZtGkTbm5ufPfddzz//PN88cUXzJw5k9jYWPbs2YOrqyuJiYmkp6dzzz33sGTJEjp37kxycvIVhd+uJzk5mT59+uR92LVp04aXXnoJYwz33nsv3333HcOGDWPChAm8+OKLjBo1ivT0dHJzc3nwwQeZOXMmI0eOJCkpiR07dujsXmVJThac2GJ11xzfaPu7NeBayao+2ecv0HqwdSG2go+dL02FSfoNgVP5nkcD3a5qswcYi9UFNAbwFpHaxphfRGQ9cBor6f/HGHPg6h2IyBRgCkDjxo2LfBCO1KJFi7yED7Bw4ULmzJlDdnY2sbGxREREXJP0K1euzLBhwwCr7PGmTZuuu+3LdWPyl0bevHlz3pl7UFAQ7dpdW3W0Vq1auLi48PDDDzNixAhGjhwJWKWHDx06lNcuKSnputU38zt//jz3338/R48evWL5jz/+yFNPPZXXHVOrVi127dpF48aN8z6IqlevftNtA3h4eDBmzJi852vXruX1118nPT2d+Ph4unTpQvfu3YmPj2fUqFGAVb8frIJujz/+OAkJCSxcuJDx48dr95CjZaTC0bVwcBUc/g7Sz/+W5PtOt8odNAwGd09HR1ph2etC7lTgPyIyCdgIxAA5ItIS8AcuD5pdIyK9jTFXZDljzCxgFli1d266p1s8Iy8p+Uv+RkZG8u6777J9+3Zq1KjBxIkTr1tiOP+FX1dXV7Kzs6+77ctnyTdrcz3u7u6EhYWxZs0avvzyS2bOnMkPP/yQ982hMBeeL3vuuecYMmQIf/zjHzly5AhDhw4t9LqX3axscuXKlfMmnLl48SKPP/44O3fupGHDhjz//PM3LdEsIkycOJHPP/+c+fPn89lnnxU5NmUHqXFweLU1pPLoeqtvvnJNaDMc2g6HFv3Bo2rB21GlojCllWOA/ANg/WzL8hhjYo0xY40xnYDnbMvOY531bzXGpBpjUoHVQA+7RF4GpaSk4O3tTbVq1Th9+vQ1k6nbQ8+ePVm8eDEAv/76KxEREde0uXDhAikpKYwcOZK3336bXbusuiMDBw68ovTw5WsBBZVNvlzG+JNPPslbPmjQID788ENycqy5PRMTEwkICODkyZPs3LkTsH4fOTk5NG3alF27dmGMISoqivDw8Ovu69KlS7i4uODj48OFCxdYsmQJYE2G4uvry/LlywHrQ+PyRPGTJ0/m9ddfp1KlSjcdGaXsLOEo/PwuzBkCb7SCZU9Ys0Z1fRAmrYSpR2DMTGvkjSb8MqUwSX8H0EpEmomIB3APsCx/AxHxEZHL2/ob1kgegJNAqIi4iYg71kXca7p3nEXnzp0JCAigbdu23H///VeUR7aXJ554gpiYGAICAnjppZcICAi4phslOTmZESNGEBQURGhoKG+99RZgDQn9+eefCQwMJCAggI8//hiwJohZvHgxnTp1uuZC7l//+lemTZuWV2r5sj/84Q/Uq1cvb/7bxYsXU6lSJRYuXMijjz5KUFAQgwcPJiMjg9DQUBo2bIi/vz9//vOf82b0ulrt2rV54IEHCAgIYNiwYXTr9lsv4meffcabb75JYGAgvXr1Ii4uDrAmnW/durVOjlIaLpyBn16D/4TA+51hzd8h6yL0/Rs88jP8aS8MfcXqwnHV0eBlVaFKK4vIcOAdrCGbc40xL4vIDCDMGLNMRMZhjdgxWN07jxljMmwjf/4L9LG99p0x5pnr78WipZVvLjs7m+zsbDw9PYmMjGTw4MFERkbi5lYx/8jS0tLo0KEDe/bswdvb+5a3o++xGzDGuiN2x2xrDH1uNjTtbZ3BtxlmzR6lygS7llY2xqwCVl217O/5Hn8FfHWd9XKAPxRmH6pwUlNTGTBgANnZ2Rhj+Oijjypswv/+++95+OGHmTZtWrESvrqO9BTY+wXsmGNVq/SsAd0esYZW1m7h6OhUMVTMbFGO1ahR44Z94hXNkCFDOHnypKPDcB4JR63hlZE/WDNK5WRa9W1GfwDtxuodsU6i3CR9Y0zeKA+l7Kmc3SBuP8ZY4+cPrbYSfaJtWK5PawiZAu3GWJUr9e/OqZSLpO/p6UlCQgK1a9fWxK/syhhDQkJC3tj/CiEny5po5Od3rBE3rpWskgfdHoFWg6BWM0dHqEpQuUj6fn5+REdH543YUMqePD098fOrAPXXM9Ng1/9gy/uQfAp8/eGOmRBwh3bdVCDlIum7u7vTrJmefSh1S5JOwJ6FsO0juJQIjbrD8Neh1RC4QYFA5bzKRdJXShVR3GE48K01zPLyFIKth0LPp6CJ094fqQpBk75SziLxGOz+3JplKt5WY8mvKwyaYZUq1r56hSZ9pcq/2N3WRdmIb63nTXpC14fAfyRUa+DY2FSZo0lfqfLIGDj+E2x+x5pasFI1uO1J6P4oeNdzdHSqDNOkr1R5kpsLB5fD5rchdhd41YWBL1p3ynoWXMpaKU36SpUHOVnw65dWso8/DDWbwch3IGiC1qZXRaJJX6myLOuSNbb+53etsfV128O4udbYehedMEYVnSZ9pcqiC2dg92ew9UNIOwd+ITDiTWg1WMsiqGLRpK9UWZGTZdXA2fmp9dPkQPN+0GeeNSJHk72yA036Sjla3GHYtQD2LIK0OPCqBz2fhI4Twaelo6NTTkaTvlKOEh8J6/4JEUvBxc26Y7bT76DlQJ15SpUYfWcpVdqSY+CnV2HXZ+DmCX2mWaWMveo4OjJVAWjSV6q0XEyETW/C9o8BAyEPQ++p4OXr6MhUBaJJX6mSlpMF2z60JhXPTLXG1vedrvPLKofQpK9USTq+CVZNhbiDVinjQTOgTltHR6UqsEIV0xaRoSJySESOiMj067zeRETWisheEdkgIn75XmssIj+IyAERiRCRpvYLX6kyKuU0LHkI5o+ErIswYRHct1gTvnK4As/0RcQV+AAYBEQDO0RkmTEmIl+zN4AFxpj5ItIfeAX4ne21BcDLxpg1IuIF5Nr1CJQqS3KyYPssWP+KNbF46F+h19PgXtnRkSkFFK57JwQ4Yow5BiAii4DRQP6kHwA8Y3u8HlhqaxsAuBlj1gAYY1LtFLdSZUt2pjU71ea3ICkKWg6CYf+G2i0cHZlSVyhM0m8InMr3PBrodlWbPcBY4F1gDOAtIrWB1sB5EfkaaAb8CEw3xuTkX1lEpgBTABo31otbqhzJSoddn/5WG6d+R5jwBbQeonfQqjLJXhdypwL/EZFJwEYgBsixbb830Ak4CXwBTALm5F/ZGDMLmAUQHBxs7BSTUiUn8yKEz4Of34PUM9Com1X1suUATfaqTCtM0o8BGuV77mdblscYE4t1po+t3/5OY8x5EYkGdufrGloKdOeqpK9UuXLiF/h6CiSfhKa9YewsaNZHk70qFwqT9HcArUSkGVayvwe4N38DEfEBEo0xucDfgLn51q0hIr7GmDigPxBmr+CVKlU52fDTv2HTG1CjCUxaCU17OToqpYqkwKRvjMkWkceB7wFXYK4xZr+IzADCjDHLgL7AKyJisLp3HrOtmyMiU4G1IiJAOPBxyRyKUiUoKQqWPAzR2yHoXhj+GlTydnRUShWZGFO2utCDg4NNWJh+GVBlyJ4vYOWfQVxg1NvQ/k5HR6TUNUQk3BgTXFA7vSNXqRtJi4fVf4V9X0HjHlbfvZZOUOWcJn2lrpabCzs/gR9fgsw06Pcc9HpGyx0rp6DvYqXyO70HVjwDMWHWyJwRb4JvG0dHpZTdaNJXCiA9Bdb/C7Z/BFVqw5hZEDheh2Eqp6NJX1Vsubnw62JY8wKknoWuD0L/56FyTUdHplSJ0KSvKq6TW+G76RC7Cxp0hgmfQ8Mujo5KqRKlSV9VPOdPWmf2+78G7wZWV06Hu8ClUJXGlSrXNOmriiMzDTa9BVvet8bch06Hnk+CR1VHR6ZUqdGkryqG2N3WpCYJkdBhPAx8Aar7FbyeUk5Gk75ybrm58Mt/YO0MqOoL9y+D5qGOjkoph9Gkr5xXymlY+ggc2wD+o2DUe1CllqOjUsqhNOkr53RwJXz7OGSnW8m+8/065l4pNOkrZ3MpCX74P2s2q/pBcOcc8Gnl6KiUKjM06SvnYIw1BHP1dLiYYE1G3vdZcPNwdGRKlSma9FX5d/6kVfo48gdo0AkmLoH6gY6OSqkySZO+Kr9ysmHbh7D+ZUBg6KsQMgVcXB0dmVJlliZ9VT4lHLXG3cfuhFZDrGqYNRoVvJ5SFZwmfVX+7PkCVj4DLm4wbi60G6sjc5QqJE36qvzIuAArp8LeRdD4NrjzY72rVqki0qSvyoeYnfDV7+H8CWtUTp+p2nev1C0oVFlBERkqIodE5IiITL/O601EZK2I7BWRDSLid9Xr1UQkWkT+Y6/AVQVhDPz8HswZBDlZMGkV9P2rJnylblGBSV9EXIEPgGFAADBBRAKuavYGsMAYEwjMAF656vV/ABuLH66qULIzYemjsOb/oM0weHQzNOnh6KiUKtcKc6YfAhwxxhwzxmQCi4DRV7UJANbZHq/P/7qIdAHqAj8UP1xVYVxMhP+NhT0Lod/zMP5Tnc1KKTsoTNJvCJzK9zzatiy/PcBY2+MxgLeI1BYRF+BNYOrNdiAiU0QkTETC4uLiChe5cl6Jx2DOYDi1DcbOhtBpOjpHKTux11RBU4FQEdkFhAIxQA7wR2CVMSb6ZisbY2YZY4KNMcG+vr52CkmVSye3weyBcDEe7v8WAu9ydERKOZXCjN6JAfLf9eJnW5bHGBOL7UxfRLyAO40x50WkB9BbRP4IeAEeIpJqjLnmYrBS7FsC3zwK1RvCvV+CT0tHR6SU0ylM0t8BtBKRZljJ/h7g3vwNRMQHSDTG5AJ/A+YCGGPuy9dmEhCsCV9d184FsOwJaNQd7vkcqtZ2dERKOaUCu3eMMdnA48D3wAFgsTFmv4jMEJHbbc36AodE5DDWRduXSyhe5Yz2LYFlT0LLgVaXjiZ8pUqMGGMcHcMVgoODTVhYmKPDUKXl0HfwxX3QqBvc9xV4VHF0REqVSyISbowJLqidvS7kKlV0xzfC4vuhXgeYsEgTvlKlQJO+coxTO+Dze6BWc5j4NXhWc3RESlUImvRV6TuzDz67E7zqwP1LdbJypUqRJn1Vus7uh0/vAPeq1kVb73qOjkipCkWTvioduTmw5X2Y1Q8QK+HXbOLoqJSqcLS0sip5icdh6R/h5BZoMwJGvWN17SilSp0mfVVyjIGd8+G7Z61SyHfMhKAJWkdHKQfSpK9KxoUz1h22kT9As1AY/YHOYatUGaBJX9lfUhR8MhLS4mHYa9D1YXDRy0dKlQWa9JV9JR6H+aOs+Wx//x006OjoiJRS+WjSV/aTeNw6w89KgweWQf0gR0eklLqKfudW9pF47LeEf78mfKXKKj3TV8WXl/Av2hJ+oKMjUkrdgCZ9VTwJR60+/KxL8MByq3iaUqrM0u4ddesunIUFo20Jf5kmfKXKAT3TV7cmK92qg38xASav1oSvVDmhSV8VnTGw/EmI3gHjP9VhmUqVI9q9o4pu89uw9wvo9zwE3F5we6VUmaFJXxXNwZWwdga0vxP6THV0NEqpItKkrwrvzK+w5GFo0MmqpaOF05QqdwqV9EVkqIgcEpEjIjL9Oq83EZG1IrJXRDaIiJ9teUcR+UVE9tteu9veB6BKSWocLJwAntXhns/BvbKjI1JK3YICk76IuAIfAMOAAGCCiARc1ewNYIExJhCYAbxiW34RuN8Y0w4YCrwjIjXsFbwqJRmp1kidtHiY8DlUq+/oiJRSt6gwZ/ohwBFjzDFjTCawCBh9VZsAYJ3t8frLrxtjDhtjIm2PY4FzgK89AlelJDUO5o+0RuqMmWl17Silyq3CJP2GwKl8z6Nty/LbA4y1PR4DeItI7fwNRCQE8ACOXr0DEZkiImEiEhYXF1fY2FVJSziGYzguAAAb/UlEQVQKcwbBuYNWl067MY6OSClVTPa6kDsVCBWRXUAoEAPkXH5RROoDnwKTjTG5V69sjJlljAk2xgT7+uoXgTIhJhzmDIb0ZKu8Qpthjo5IKWUHhbk5KwbIP+WRn21ZHlvXzVgAEfEC7jTGnLc9rwasBJ4zxmy1R9CqhEWugcX3Q1UfmPg1+LRydERKKTspzJn+DqCViDQTEQ/gHmBZ/gYi4iMil7f1N2CubbkH8A3WRd6v7Be2KjG7/gef3w21W8KDP2rCV8rJFJj0jTHZwOPA98ABYLExZr+IzBCRy7dj9gUOichhoC7wsm35eKAPMElEdtv+6T37ZdXOT+Hbx6BZb5i0ErzrOjoipZSdiTHG0TFcITg42ISFhTk6jIrnxC9WieSmveDexeDm4eiIlFJFICLhxpjggtrpHbkKzp+ELyZCzSZw1zxN+Eo5MU36FV1GqnWnbU4WTFgElWs6OiKlVAnS0soVWW4ufPMHOBcB932pF22VqgA06VdkG16BgytgyL+g5UBHR6OUKgXavVNR7fsaNr4GHSdC9z86OhqlVCnRpF8Rxe6GpX+ERt1g5FtaIlmpCkSTfkVzMRG++B1UqQV3/w/cKjk6IqVUKdI+/YokNwe+fhhSz8Dk78CrjqMjUkqVMk36FclPr8GRH2Hk2+DXxdHRKKUcQLt3KorINfDTvyHoXugy2dHRKKUcRJN+RZAUBUsegrrtYcSbeuFWqQpMk76zy0q3yiQbA3cvAI8qjo5IKeVA2qfv7FZNhdN7YMIXUKu5o6NRSjmYnuk7s50LYNen0HsqtBnq6GiUUmWAJn1nFR0GK/8MzftBv2cdHY1SqozQpO+MLpy1SiV714dxc8HF1dERKaXKCO3TdzbZmbD4d9aE5g+use68VUopG036zmb1X+DUNhg3D+q1d3Q0SpU7xhjEiYc1a9J3JmHzIHwe9Hoa2o91dDRKlSv7Y5N5atFuki5mckfHhtzZxQ//+tUcHZbdadJ3Fie3wappVl38/v/n6GiUKjeMMfxv20n+sSKCWlU86Ny4JvN/iWL25uO0a1CNOzv7MbpjA2p7OUdxwkIlfREZCrwLuAKzjTGvXvV6E2Au4AskAhONMdG21x4Anrc1/acxZr6dYleXpZy2+vGr+8Gds/XCrSqXcnINPx+JZ+nuGFxFCG3jS6+WPtSocv05m9OzcgiLSuKnw+c4ci6VFr5eBDSoRkCDarTw9cLdteBxKinpWfzt619Zufc0oa19eWt8ELW9KpGYlsmy3TEs2RnDjBUR/GvVAYKb1qRB9cr4elfC17sSPl7WzwY1KtO0dpVy0yUkxpibNxBxBQ4Dg4BoYAcwwRgTka/Nl8AKY8x8EekPTDbG/E5EagFhQDBggHCgizEm6Ub7Cw4ONmFhYcU8rAokNwfmDoWz++GhH6FugKMjUqXAmfqdj5xLZcnOaL7ZGcOZlHSqeVrnoinp2bgIBDWqQWhrX/q09qV6ZXc2HY7jp8Nx/HIsgfSsXDxcXWjmU5WohDQysnMB8HB1oXU9L/zrWR8CAfWr4d+gGtU83fP2+2t0Mo8v3El00iWmDm7DH/o0x8Xl2t/pwTMpLAmPZkdUEnEXMohLzSDTtp/L6lXzzIuxV0sfqldxv2Y7N5Kba4g4ncJPh+NwEeHRvi1u5deIiIQbY4ILbFeIpN8DeNEYM8T2/G8AxphX8rXZDww1xpwS652YbIypJiITgL7GmD/Y2n0EbDDGLLzR/jTpF9GO2dZ4/LEfQ+B4R0ejSlBGdg5Ld8Uwa+MxTiVdwtd2ppn/rLO5T1UGBtTFq1LBX+IT0zLZeDiOgAbVaF3X+5bjMsawPzaFsKhEbp5NrnQpK4cf9p9l96nzuLoIoa19GdfFjwH+dXAVYU90MhttCX5P9Hnyp6pmPlXp08qH0Da+dG9emyoebmTn5HI8Po2I0ylExKYQcTqF/bEpJKZl5q3XqFZlAupXo241TxZtP0VtLw/en9CJ4KaFH+VmjCElPZv41AziLmRwPD6NTZFxbI6Mz/ug6tioBr1b+dLct+oV/0/VK7sjIiSkZrD5SDw/HYpjY2Qc8alWjKGtfZn/+5Ai/BZ/Y8+kPw4roT9ke/47oJsx5vF8bT4Hthlj3hWRscASwAeYDHgaY/5pa/d/wCVjzBtX7WMKMAWgcePGXU6cOFH4I63I0hLg/c5QrwM8sFwLqTmp1IxsPt92gjmbj3M2JYN2DarRs6VPXtKJu5BBfGoGCWmZGAOV3V0Z1r4e47r40b157SvOXrNycll/8BxLdkaz7uA5snKsv/8gv+rc2cWP24Ma3LA75WpxFzL4dncMX4VHc/DMhVs6tjZ1vRnXxY/RnRpQx9vzhu2S0jLZfCSeC+nZ9GxZmya1qxZq+8YYzl3IuOKD4EBsCscT0hjQtg6vjwuiZtXCHW9BsnNy2RN9np8OWR9Ue2OSuTq9ursKtap6cO5CBsZAzSru9GntS59WvvRu7XPT30FBSjvpNwD+AzQDNgJ3Au2BhyhE0s9Pz/SLYNmTsPszeGQz1PF3dDRO52TCRWZtOoqbi0teF0Grul5Uciv6NRNjDOcvZuHl6VaovuacXMOZlHQWbjvJgl+iSEnP5rYWtXm0bwt6tfS5btfO5aTzVXgMK/bEciEjm4Y1KjO2c0N6NK/NjwfO8e3uGBLSMvHxqsSYTg0Y2r4+u0+d56vwaA6cTsHD1YUB/nUY18WPgAbXjlwxBvbY2m84HEdOriGoUQ3GdfFjkH9dPN0Lf7+nIFSr7OaQbqqsnNxC/T8Ux4X0LM6mZOR1CeX/gG5cqwqhrX1p37A6rtfpUroVpdq9c1V7L+CgMcZPu3dKUEw4fDwAejwGQ152dDROJfliFv9ZH8n8LScQARcRLmXlAODmIrSs40VA/WqEtvFlRIf6uBWQPCJiU3h5VQQ/H0kAoFZVj7yv/D5eHtSs6sGF9Oy8pBCXmkFiWiY5uQYRGNquHo+EtiCoUY1CH0N6Vg4/RJzlq/BoNkfGkWusfu6BAXW4s7MffVr7XpP09scmsyQ8hqW7Y67oErmeutUqMaaTH+O6NKRlnVvvGlL2Y8+k74Z1IXcAEIN1IfdeY8z+fG18gERjTK6IvAzkGGP+bruQGw50tjXdiXUhN/FG+9OkXwi5uTBnIJw/BU+Eg6fzjSUuCedS0tl5Monmvl4096l6TbLOysnls60neGdtJMmXshjX2Y+pQ9rg41WJEwm/9RUfOJ3CvtgU4i5k0LBGZab0ac744EZU9nC9Zn9v/HCIL8OjqV7ZnQd6NMVFhLjU9CsTfGom1Sq7W/2+V/XT92rlQwtfr2Id95nkdHadTKJ789qF6srIysllc2Q8Z1PSr/t6w5qVua2Fj93OUJV92C3p2zY2HHgHa8jmXGPMyyIyAwgzxiyzdQG9gjVCZyPwmDEmw7bu74HLFb9eNsbMu9m+NOkXws4FsOwJGPMRBN3j6GjKvOPxaczaeJQl4TFk5thGd7i50LaetzWqo341vCq58cH6IxyLT6Nny9o8O9yfdg2q33CbubmG9YfOMXPDUcJOJFGrqgeTbmvK/T2a4OHmwscbj/PRxqNk5eTyQI+mPNG/VZFGdChVVHZN+qVJk34BLiXB+13ApzVMXq0Xb29ib/R5PvzpKKv3ncHd1YW7uvhxR6eGRCddzLuoFxGbQtLFLABa+FbluRH+9GtTp0j9zDuiEvlww1HWHjxHFQ9XvCq5ce5CBsPa12P6sLaFvuioVHEUNunrHbnlzbqXrcQ/7DVN+Dew7VgC766NZMvRBLw93Xg0tAWTezbD19u6o7Jr01qM6WS1Nca6YBqTdImgRjVu6eJe16a16DqpFofOXOCjjUeJT83kg/ta0rUIwwCVKi2a9MuT03shbA4EPwj1Ax0dTZlzLC6VV1YfZE3EWep4V+LZ4W2ZENIYb88bd6uICPWrV6Z+9crF3n+bet68Nb5jsbejVEnSpF9eGGPV1qlcE/o/5+hoypSktEzeXRvJ/7aeoJKbC9OGtOHBXs3wdNdyFEpdTZN+eXFgGZzaCqPesxK/IiM7h09/OcF7ayNJzcjm7q6NeWZQ67xuHKXUtTTplwe5ObD+X9bF204THR2Nwx2NS2VJeDRf22q19Gnty3PD/WlTT8eLK1UQTfrlwb4lEHcQ7vqkwlbQTL6UxYq9sXwVHs2uk1atlj6tfPj3uEBCW/s6Ojylyg1N+mVdThZseAXqdgD/0Y6OplSdS0lnY2Q86w6e5ccD58jMzqVNXW+eG+5fYK0WpdT1adIv6/YshMRjMGERuDj3PPaZ2bmEnUhk4+F4fjocx4HTKQD4elfi3pDGjOviR7sG1ZympLBSjqBJvyzLzoCfXoOGXaD1UEdHU2Jiz19i9qbjLA47RWpGNm4uQnDTmvx1aFtCW/viX99bE71SdqJJvyzbuQCST8Ht7znljViRZy/w4U/H+HZ3DAYYFVif4R3qc1tLn0LVg1dKFZ3+ZZVVmRdh4+vQpCc07+foaOwq/EQiMzcc48cDZ6ns7srE7k14qHcz/GpWcXRoSjk9TfplVdgcSD0L4+aV27P87JxcohLS2J+vzs2B0xeIT82gRhV3/jSgFQ/c1pRadprEQilVME36ZVHGBdj8NrToD017OjqaIrmUmcMPEWdYsjOGbccS8uYsdXcVWtXxpm8bXzo3rskdnRpQxUPffkqVNv2rK4u2fQgXE6Df846OpFCMMYSfSGLJzmhW7DmdN2PTvd0a075BdQIaVKOFrxcebs49+kip8kCTfllzKQl+fh/aDAe/Lo6O5qaycnKZvyWKz7ad5Hh8mjU3awfb3KzNrpybVSlVNmjSL2u2fggZydDv2YLbOtC+mGT+8tVeIk6nENKsFn/s24JhHerrqBulyjj9Cy1LMi5YXTttRkC9Do6O5rrSs3J4b20kH208Rq2qHnz0uy4MaVfP0WEppQpJk35ZEjYX0s9D7z87OpLrCj+RyF++2svRuDTu6uLH8yMCdApApcoZTfplRdYl2PIfaN63zPXln0hIY+7m4yzYeoIG1Ssz//chWuRMqXJKk35Zset/kHYOes91dCQApGZks2rvab4Kj2Z7VCIiMLFbE/46rK322ytVjhXqr1dEhgLvAq7AbGPMq1e93hiYD9SwtZlujFklIu7AbKCzbV8LjDGv2DF+55CTBT+/C426QdNeDgsjN9fwy7EEvgqPZvW+06Rn5dLcpyrThrRhTKeGNKhR/CkFlVKOVWDSFxFX4ANgEBAN7BCRZcaYiHzNngcWG2NmikgAsApoCtwFVDLGdBCRKkCEiCw0xkTZ+TjKt72LrRo7I95yyN23WTm5LN8Ty4c/HeXw2VS8Pd0Y29mPcV386NSohhY7U8qJFOZMPwQ4Yow5BiAii4DRQP6kb4BqtsfVgdh8y6uKiBtQGcgEUuwQt/PIzYHNb1mjdVoNKtVdX8zM5osdp5i96Tgx5y/Rpq43b94VxIjA+jq/rFJOqjBJvyFwKt/zaKDbVW1eBH4QkSeAqsBA2/KvsD4gTgNVgKeNMYlX70BEpgBTABo3blyE8J1AxLeQcMSaFauUzqiTL2Ux7+fjzN8SRdLFLLo2rck/7mhHvzZ19KxeKSdnrytyE4BPjDFvikgP4FMRaY/1LSEHaADUBDaJyI+XvzVcZoyZBcwCCA4ONnaKqewzBja9BbVbgf/tpbLL7/ef4fml+4i7kMFA/7o82rc5XZrUKpV9K6UcrzBJPwZolO+5n21Zfg8CQwGMMb+IiCfgA9wLfGeMyQLOicjPQDBwDAWRP8DZX+GOmSU+9218agYvLNvPyr2n8a9fjbkPdKWDX/US3adSquwpTAWsHUArEWkmIh7APcCyq9qcBAYAiIg/4AnE2Zb3ty2vCnQHDton9HLOGNj4BlRvDB3uKsHdGL7dHcOgt35izf6zTB3cmmWP99SEr1QFVeCZvjEmW0QeB77HGo451xizX0RmAGHGmGXAn4GPReRprIu3k4wxRkQ+AOaJyH5AgHnGmL0ldjTlSdQmiN4Ow98A15K5q/V08iWe/2Yfaw+eo2OjGrw+LpBWdb1LZF9KqfJBjClbXejBwcEmLCzM0WGUvP+Ng9N74Klfwd3Trps+Hp/GrI1HWRIeg4sLTB3chsk9m+GqVS+VcloiEm6MCS6ond5a6QgJR+HIGuj7N7sm/L3R5/nwp6Os3ncGd1cX7gr245HQFjSqpdMQKqUsmvQdYccccHGDLpPssrntxxN5d+1hfj6SgLenG3/s24JJtzXD17uSXbavlHIemvRLW2aaVWcnYDR4F78k8ZFzqdw3eys1q3jw7PC2TAhpjLenVr5USl2fJv3StvcLa5KUkD8Ue1PGGF5ctp/K7q6s+lNvfLz0zF4pdXM6aWlpMga2fwz1AqFRSLE3992+M2w+Es+fB7fRhK+UKhRN+qXpxM9wLgJCphS75MKlzBz+sSKCtvW8ua9bBStdoZS6ZZr0S9P2WVC5JnQYV+xN/XfDEWKT0/nHHe1xc9X/RqVU4Wi2KC3J0XBgBXS+H9yLV5c+Kj6Nj346xphODenaVOvmKKUKT5N+aQmbByYXgh8s9qZmrIjAw82Fvw1ra4fAlFIViSb90pCdAeGfQJthULNJsTb1Y8RZ1h08x1MDW1Gnmn3v5FVKOT9N+qVh/1K4GA8hDxdrM+lZOby0Yj8t63jxwG1N7RObUqpC0XH6pWH7R1bN/Ob9irWZj346xqnES3z+UDfc9eKtUuoWaOYoadHhEBNe7GGaJxLS+O+GI4wIrM9tLX3sGKBSqiLRpF/SdnwMHl4QdM8tbyI31zDtq714uLrw/Ah/OwanlKpoNOmXpKxLELEM2t8JntUKbn8D83+JYvvxRP5vVAD1qxdvuKdSqmLTpF+Sjm2ArDSruNotiopP49/fHaRfG1/u6uJnv9iUUhWSJv2SdGA5VKoOTXvf0upWt84e3F1deGVsIFLM0g1KKaVJv6TkZMOh1dBmKLh53NIm5m2JYkdUEi+Make96jomXylVfJr0S8rJLXApEdqOvKXVj8en8fr3B+nftg53dm5o5+CUUhWVJv2ScmA5uHlCywFFXjUn1zDtyz14uLrwytgO2q2jlLKbQiV9ERkqIodE5IiITL/O641FZL2I7BKRvSIyPN9rgSLyi4jsF5FfRcT5+ylyc63iai0HgkfVIq8+7+fjhJ1I4sXb21FXSy0opeyowKQvIq7AB8AwIACYICIBVzV7HlhsjOkE3AP817auG/A/4BFjTDugL5Blt+jLqthdcCH2lrp2rG6dQwz0r8OYTtqto5Syr8Kc6YcAR4wxx4wxmcAi4OoxiAa4PBC9OhBrezwY2GuM2QNgjEkwxuQUP+wy7uBya+Lz1kOKvOrczccB+NcY7dZRStlfYZJ+Q+BUvufRtmX5vQhMFJFoYBXwhG15a8CIyPcislNE/nK9HYjIFBEJE5GwuLi4Ih1AmWOM1Z/ftBdUKVqt+/SsHL7dHcPQ9vW0gqZSqkTY60LuBOATY4wfMBz4VERcsAq69QLus/0cIyLXXNk0xswyxgQbY4J9fX3tFJKDxB2ChCPgP6rIq36//wwp6dmMD25UAoEppVThkn4MkD8L+dmW5fcgsBjAGPML4An4YH0r2GiMiTfGXMT6FtC5uEGXaQeXWz/bjCjyql+GRdOwRmV6NK9t56CUUspSmKS/A2glIs1ExAPrQu2yq9qcBAYAiIg/VtKPA74HOohIFdtF3VAgwl7Bl0kHloNfCFSrX6TVopMu8vPReO4K9sPFRfvylVIlo8Ckb4zJBh7HSuAHsEbp7BeRGSJyu63Zn4GHRWQPsBCYZCxJwFtYHxy7gZ3GmJUlcSBlwvmTcHoP+Bd91M6ScOvL0zitr6OUKkGFmkTFGLMKq2sm/7K/53scAfS8wbr/wxq26fwOrLB+FnGoZm6u4cvwU/Rs4YNfzSolEJhSSln0jlx7OrgC6rSD2i2KtNrWYwlEJ13irmA9y1dKlSxN+vaSGgcnttxS187isFNU83RjSLt6JRCYUkr9RpO+vRxaBZgiD9VMvpTF6n1nGN2xIZ7uriUTm1JK2WjSt5cDy6FGE6jbvkirLd8TS0Z2ro7NV0qVCk369pCeAsd/ss7yi1g64cuwU7St5037hrc+naJSShWWJn17iPwBcjKL3LVz8EwKe6KTGR/cSOvsKKVKhSZ9eziwHKrWsW7KKoIvw6JxdxXu0GqaSqlSokm/uLLS4ciP0HY4uBT+15mZncs3u2IYFFCXWlVvbTpFpZQqKk36xXVsA2SmQtuide2sO3iWxLRM7uqiF3CVUqVHjDGOjuEKIhIHnCjGJnyAeDuFU57ocVcsetwVS2GOu4kxpsAyxWUu6ReXiIQZY4IdHUdp0+OuWPS4KxZ7Hrd27yilVAWiSV8ppSoQZ0z6sxwdgIPocVcsetwVi92O2+n69JVSSt2YM57pK6WUugFN+kopVYE4TdIXkaEickhEjojIdEfHU5JEZK6InBORffmW1RKRNSISaftZ05Ex2puINBKR9SISISL7ReRPtuXOftyeIrJdRPbYjvsl2/JmIrLN9n7/wjZ/tdMREVcR2SUiK2zPK8pxR4nIryKyW0TCbMvs8l53iqQvIq7AB8AwIACYICIBjo2qRH0CDL1q2XRgrTGmFbDW9tyZZAN/NsYEAN2Bx2z/x85+3BlAf2NMENARGCoi3YF/A28bY1oCScCDDoyxJP0Ja27uyyrKcQP0M8Z0zDc+3y7vdadI+kAIcMQYc8wYkwksAkY7OKYSY4zZCCRetXg0MN/2eD5wR6kGVcKMMaeNMTttjy9gJYKGOP9xG2NMqu2pu+2fAfoDX9mWO91xA4iIHzACmG17LlSA474Ju7zXnSXpNwRO5XsebVtWkdQ1xpy2PT4D1HVkMCVJRJoCnYBtVIDjtnVx7AbOAWuAo8B5Y0y2rYmzvt/fAf4C5Nqe16ZiHDdYH+w/iEi4iEyxLbPLe93NHtGpssUYY0TEKcfiiogXsAR4yhiTkn8eAmc9bmNMDtBRRGoA3wBtHRxSiRORkcA5Y0y4iPR1dDwO0MsYEyMidYA1InIw/4vFea87y5l+DJC/XKWfbVlFclZE6gPYfp5zcDx2JyLuWAn/M2PM17bFTn/clxljzgPrgR5ADRG5fNLmjO/3nsDtIhKF1V3bH3gX5z9uAIwxMbaf57A+6EOw03vdWZL+DqCV7cq+B3APsMzBMZW2ZcADtscPAN86MBa7s/XnzgEOGGPeyveSsx+3r+0MHxGpDAzCup6xHhhna+Z0x22M+Zsxxs8Y0xTr73mdMeY+nPy4AUSkqoh4X34MDAb2Yaf3utPckSsiw7H6AF2BucaYlx0cUokRkYVAX6xyq2eBF4ClwGKgMVZp6vHGmKsv9pZbItIL2AT8ym99vM9i9es783EHYl20c8U6SVtsjJkhIs2xzoBrAbuAicaYDMdFWnJs3TtTjTEjK8Jx247xG9tTN+BzY8zLIlIbO7zXnSbpK6WUKpizdO8opZQqBE36SilVgWjSV0qpCkSTvlJKVSCa9JVSqgLRpK+UUhWIJn2llKpA/h8jFx6GsPpecgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f1463828ba8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "print(\"History:\",hist.history[\"val_acc\"])\n",
    "print(\"Max accuracy:\",numpy.max(hist.history[\"val_acc\"]))\n",
    "plt.ylim(0.85,1.0)\n",
    "plt.plot(hist.history[\"val_acc\"],label=\"Validation set accuracy\")\n",
    "plt.plot(hist.history[\"acc\"],label=\"Training set accuracy\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}