rnn_text_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RNN text classification\n",
    "\n",
    "In this notebook, we'll go through a straightforward application of recurrent neural networks to binary text classification in Keras."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load IMDB data\n",
    "\n",
    "This is the same data as in [the BoW classifier notebook](https://github.com/TurkuNLP/Deep_Learning_in_LangTech_course/blob/master/bow_classifier.ipynb).\n",
    "\n",
    "(Note that there is direct support for IMDB in keras (`keras.datasets.imdb`). We're doing the preprocessing explicitly here to show all the steps.)\n",
    "\n",
    "First, download the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File ‘imdb_train.json’ already there; not retrieving.\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "# Give -nc (--no-clobber) argument so that the file isn't downloaded multiple times \n",
    "!wget -nc https://raw.githubusercontent.com/TurkuNLP/Deep_Learning_in_LangTech_course/master/data/imdb_train.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll load the JSON-formatted data. This gives us a list of dictionaries, each of which has `class` and `text`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'list'>\n",
      "<class 'dict'>\n",
      "dict_keys(['class', 'text'])\n",
      "{'class': 'pos', 'text': \"With all this stuff going down at the moment with MJ i've started listening to his music, watching the odd documentary here and there, watched The Wiz and watched Moonwalker again. Maybe i just want to get a certain insight into this guy who i thought was really cool in the eighties just to maybe make up my mind whether he is guilty or innocent. Moonwalker is part biography, part feature film which i remember going to see at the cinema when it was originally released. Some of it has subtle messages about MJ's feeling towards the press and also the obvious message of drugs are bad m'kay.  Visually impressive but of course this is all about Michael Jackson so unless you remotely like MJ in anyway then you are going to hate this and find it boring. Some may call MJ an egotist for consenting to the making of this movie BUT MJ and most of his fans would say that he made it for the fans which if true is really nice of him.  The actual feature film bit when it finally starts is only on for 20 minutes or so excluding the Smooth Criminal sequence and Joe Pesci is convincing as a psychopathic all powerful drug lord. Why he wants MJ dead so bad is beyond me. Because MJ overheard his plans? Nah, Joe Pesci's character ranted that he wanted people to know it is he who is supplying drugs etc so i dunno, maybe he just hates MJ's music.  Lots of cool things in this like MJ turning into a car and a robot and the whole Speed Demon sequence. Also, the director must have had the patience of a saint when it came to filming the kiddy Bad sequence as usually directors hate working with one kid let alone a whole bunch of them performing a complex dance scene.  Bottom line, this movie is for people who like MJ on one level or another (which i think is most people). If not, then stay away. It does try and give off a wholesome message and ironically MJ's bestest buddy in this movie is a girl! Michael Jackson is truly one of the most talented people ever to grace this planet but is he guilty? Well, with all the attention i've gave this subject....hmmm well i don't know because people can be different behind closed doors, i know this for a fact. He is either an extremely nice but stupid guy or one of the most sickest liars. I hope he is not the latter.\"}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "\n",
    "with open('imdb_train.json') as f:\n",
    "    data = json.load(f)\n",
    "    \n",
    "\n",
    "# Look at the data\n",
    "print(type(data))\n",
    "print(type(data[0]))\n",
    "print(data[0].keys())\n",
    "print(data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's be careful and shuffle our data first, and then separate the texts and labels for further processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text: I saw Insomniac's Nightmare not to long ago for the first time and I have to say, I really found it to be quite good. If you are a fan of Dominic Monaghan you will love it. The hole movie takes place inside his mind -or does it? The acting from everyone else is a little rushed and shaky and some of the scenes could be cut down but it works out in the end. The extras on the DVD are just as great as the film, if not greater for those Dom fans. It has tons of candid moments from the set, outtakes and a great interview with the director. Anyone who has gone through making an independent film will love to watch Tess (the director), Dom and everyone else on the very small close personal set try to bang out this little trippy creepy film. It was pretty enjoyable and I'm glad to have it in my collection.\n",
      "Label: pos\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "import numpy as np\n",
    "\n",
    "    \n",
    "random.seed(1234)    # This makes the shuffle produce the same order every time\n",
    "random.shuffle(data)\n",
    "\n",
    "texts = [example['text'] for example in data]\n",
    "labels = [example['class'] for example in data]\n",
    "\n",
    "# Example text and label\n",
    "print('Text:', texts[0])\n",
    "print('Label:', labels[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess data\n",
    "\n",
    "Both the text and the labels are strings; we'll need to convert the labels into integers and the text into an appropriate format for RNN training.\n",
    "\n",
    "### Labels\n",
    "\n",
    "For the labels, we'll use the simple sklearn [`LabelEncoder`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html), as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'list'> ['pos', 'neg', 'pos', 'pos', 'pos', 'neg', 'neg', 'pos', 'neg', 'pos']\n",
      "<class 'numpy.ndarray'> [1 0 1 1 1 0 0 1 0 1]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "\n",
    "label_encoder = LabelEncoder()    # Turns class labels into integers\n",
    "Y = label_encoder.fit_transform(labels)\n",
    "\n",
    "# See how those compare\n",
    "print(type(labels), labels[:10])\n",
    "print(type(Y), Y[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Texts\n",
    "\n",
    "We'll then use a keras [`Tokenizer`](https://keras.io/preprocessing/text/) to vectorize the texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "\n",
    "\n",
    "# Excepting for num_words, the values below are the defaults for Tokenizer parameters.\n",
    "# They are included here explicitly to show some of the options for using Tokenizer.\n",
    "\n",
    "tokenizer = Tokenizer(\n",
    "    num_words=100000,\n",
    "    lower=True,\n",
    "    split=' ',\n",
    "    char_level=False\n",
    ")\n",
    "\n",
    "tokenizer.fit_on_texts(texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`tokenizer.fit_on_texts()` creates a mapping from tokens to integers.\n",
    "\n",
    "Note: that the value 0 is not used in the mapping. This allows us to use it for padding. (There isn't anything magical about 0, this is just convention.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None\n",
      "{'a': 3,\n",
      " 'and': 2,\n",
      " 'i': 9,\n",
      " 'in': 7,\n",
      " 'is': 6,\n",
      " 'it': 8,\n",
      " 'of': 4,\n",
      " 'the': 1,\n",
      " 'this': 10,\n",
      " 'to': 5}\n"
     ]
    }
   ],
   "source": [
    "from pprint import pprint    # pretty-printer\n",
    "\n",
    "\n",
    "def truncate_dict(d, count=10):\n",
    "    # Returns at most count items from the given dictionary.  \n",
    "    return dict(i for i, _ in zip(d.items(), range(count)))\n",
    "\n",
    "\n",
    "# Check if 0 is in the index, and print examples of the mapping\n",
    "print(tokenizer.word_index.get(0))\n",
    "pprint(truncate_dict(tokenizer.word_index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After initializing the token string to integer mapping, we use `texts_to_sequences` to convert each text to a corresponding sequence of integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text: I saw Insomniac's Nightmare not to long ago for the first time and I have to say, I really found it \n",
      "Sequence: [9, 215, 20597, 1719, 20, 5, 192, 592, 14, 1, 82, 54, 2, 9, 24, 5, 131, 9, 62, 254, 8, 5, 26, 175, 48, 43, 21, 22, 3, 333, 4, 7261, 13419, 21, 76, 115, 8, 1, 2931, 16, 300, 269, 1000, 23, 326, 38, 123, 8, 1, 112, 35, 312, 330, 6, 3, 113, 3308, 2, 5073, 2, 45, 4, 1, 135, 96, 26, 601, 176, 17, 8, 491, 42, 7, 1, 126, 1, 2252, 19, 1, 284, 22, 39, 13, 83, 13, 1, 18, 43, 20, 2780, 14, 144, 8057, 447, 8, 44, 3395, 4, 14895, 384, 35, 1, 266, 13420, 2, 3, 83, 2695, 15, 1, 163, 255, 33, 44, 820, 139, 227, 31, 1720, 18, 76, 115, 5, 102, 8058, 1, 163, 8057, 2, 312, 330, 19, 1, 51, 388, 487, 961, 266, 349, 5, 4009, 42, 10, 113, 12257, 943, 18, 8, 12, 180, 733, 2, 142, 1259, 5, 24, 8, 7, 57, 1587]\n",
      "Mapped back: ['i', 'saw', \"insomniac's\", 'nightmare', 'not', 'to', 'long', 'ago', 'for', 'the']\n"
     ]
    }
   ],
   "source": [
    "sequences = tokenizer.texts_to_sequences(texts)\n",
    "\n",
    "# Print an example text, its corresponding sequence, and the tokens it represents\n",
    "print('Text:', texts[0][:100])\n",
    "print('Sequence:', sequences[0])\n",
    "print('Mapped back:', [tokenizer.index_word[i] for i in sequences[0][:10]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that these sequences are variable length:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Lengths: [160, 217, 182, 270, 252, 220, 335, 117, 172, 283] min: 10 max: 2473 mean: 233.63884\n"
     ]
    }
   ],
   "source": [
    "lengths = [len(s) for s in sequences]\n",
    "print('Lengths:', lengths[:10], 'min:', min(lengths), 'max:', max(lengths), 'mean:', np.mean(lengths))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pad and truncate sequences\n",
    "\n",
    "RNNs can process variable-length inputs. However, RNN implementations can have their own constraints.\n",
    "\n",
    "Keras expects all examples in a batch to be the same length. To make it possible to train with batch sizes larger than one, we use [`pad_sequences`](https://keras.io/preprocessing/sequence/#pad_sequences) to pad and truncate all sequences to a fixed length.\n",
    "\n",
    "(An alternative approach is to use `batch_size=1` when training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "type(sequences): <class 'list'>\n",
      "type(padded): <class 'numpy.ndarray'>\n",
      "padded.shape: (25000, 250)\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "\n",
    "# The padding and truncating parameters default to 'pre', i.e. aligning\n",
    "# the end of the input with the end of the padded sequence. With 'post',\n",
    "# we're here aligning the beginning instead.\n",
    "\n",
    "post_padded_X = pad_sequences(\n",
    "    sequences,\n",
    "    maxlen=250,\n",
    "    padding='post',\n",
    "    truncating='post',\n",
    "    value=0\n",
    ")\n",
    "\n",
    "# Compare those\n",
    "print('type(sequences):', type(sequences))\n",
    "print('type(padded):', type(post_padded_X))\n",
    "print('padded.shape:', post_padded_X.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**This is the representation of the data that our model will see.**\n",
    "\n",
    "As in the case of the bag-of-words MLP example, we have produced a matrix of shape `(num_examples, n)`.\n",
    "\n",
    "However, these are quite different: in the bag-of-words representation `n` was the size of the vocabulary and each row consisted of indicator values (0 or 1) signaling the presence or absence of a word in a document. Here `n` is the (padded) sequence length, and the values are the indices of vocabulary items."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Post-padded:\n",
      "[    9   215 20597  1719    20     5   192   592    14     1    82    54\n",
      "     2     9    24     5   131     9    62   254     8     5    26   175\n",
      "    48    43    21    22     3   333     4  7261 13419    21    76   115\n",
      "     8     1  2931    16   300   269  1000    23   326    38   123     8\n",
      "     1   112    35   312   330     6     3   113  3308     2  5073     2\n",
      "    45     4     1   135    96    26   601   176    17     8   491    42\n",
      "     7     1   126     1  2252    19     1   284    22    39    13    83\n",
      "    13     1    18    43    20  2780    14   144  8057   447     8    44\n",
      "  3395     4 14895   384    35     1   266 13420     2     3    83  2695\n",
      "    15     1   163   255    33    44   820   139   227    31  1720    18\n",
      "    76   115     5   102  8058     1   163  8057     2   312   330    19\n",
      "     1    51   388   487   961   266   349     5  4009    42    10   113\n",
      " 12257   943    18     8    12   180   733     2   142  1259     5    24\n",
      "     8     7    57  1587     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0]\n",
      "Mapped back: ['i', 'saw', \"insomniac's\", 'nightmare', 'not', 'to', 'long', 'ago', 'for', 'the']\n"
     ]
    }
   ],
   "source": [
    "print('Post-padded:')\n",
    "print(post_padded_X[0])\n",
    "print('Mapped back:', [tokenizer.index_word[i] for i in post_padded_X[0][:10]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build model\n",
    "\n",
    "We define a basic RNN model that takes the RNN cell class (`RNN_class`) as an argument:\n",
    "\n",
    "* input: sequence of `sequence_length` integers corresponding to words\n",
    "* embedding: randomly initialized mapping to `embedding_dim`-dimensional vectors\n",
    "* rnn: recurrent neural network with `rnn_units`-dimensional state\n",
    "* output: `num_classes`-dimensional fully connected layer with softmax activation\n",
    "\n",
    "We're intentionally leaving out a few fairly obvious things that would be expected to help here, including\n",
    "\n",
    "* Any form of regularization, e.g. dropout\n",
    "* Initializing the embeddings with pre-trained word vectors\n",
    "* Masking to ignore padding (see [Masking and padding with Keras](https://www.tensorflow.org/guide/keras/masking_and_padding))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/smp/Library/Python/3.7/lib/python/site-packages/tensorflow_core/python/keras/initializers.py:119: calling RandomUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
      "WARNING:tensorflow:From /Users/smp/Library/Python/3.7/lib/python/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.layers import Input, Embedding, SimpleRNN, LSTM, Dense\n",
    "\n",
    "\n",
    "def build_rnn_model(RNN_class, sequence_length, vocab_size, num_classes,\n",
    "                    embedding_dim=250, rnn_units=50):\n",
    "    input_ = Input(shape=(sequence_length,))\n",
    "    embedding = Embedding(vocab_size, embedding_dim)(input_)\n",
    "    # return_sequences=False is the default\n",
    "    rnn = RNN_class(rnn_units, return_sequences=False)(embedding)\n",
    "    output = Dense(num_classes, activation='softmax')(rnn)\n",
    "    return Model(inputs=[input_], outputs=[output])\n",
    "\n",
    "\n",
    "sequence_length = post_padded_X.shape[1]\n",
    "vocab_size = tokenizer.num_words\n",
    "num_classes = len(label_encoder.classes_)\n",
    "\n",
    "# We'll use the defaults for embedding_dim and rnn_units in all of our examples here.\n",
    "simple_rnn_model = build_rnn_model(SimpleRNN, sequence_length, vocab_size, num_classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         [(None, 250)]             0         \n",
      "_________________________________________________________________\n",
      "embedding (Embedding)        (None, 250, 250)          25000000  \n",
      "_________________________________________________________________\n",
      "simple_rnn (SimpleRNN)       (None, 50)                15050     \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 2)                 102       \n",
      "=================================================================\n",
      "Total params: 25,015,152\n",
      "Trainable params: 25,015,152\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "simple_rnn_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train simple RNN\n",
    "\n",
    "We'll then compile and train our model.\n",
    "\n",
    "Note that as our outputs `Y` are integers (rather than one-hot class indicators), we need to use `sparse_categorical_crossentropy` (instead of `categorical_crossentropy`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "simple_rnn_model.compile(\n",
    "    loss='sparse_categorical_crossentropy',\n",
    "    optimizer='adam',\n",
    "    metrics=['accuracy']\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 22500 samples, validate on 2500 samples\n",
      "Epoch 1/3\n",
      "22500/22500 [==============================] - 629s 28ms/sample - loss: 0.6986 - acc: 0.5019 - val_loss: 0.6949 - val_acc: 0.5032\n",
      "Epoch 2/3\n",
      "22500/22500 [==============================] - 749s 33ms/sample - loss: 0.6494 - acc: 0.5784 - val_loss: 0.7135 - val_acc: 0.4916\n",
      "Epoch 3/3\n",
      "22500/22500 [==============================] - 761s 34ms/sample - loss: 0.6409 - acc: 0.5839 - val_loss: 0.7159 - val_acc: 0.5076\n"
     ]
    }
   ],
   "source": [
    "# We'll use these parameters for training in these examples\n",
    "epochs = 3\n",
    "batch_size = 16\n",
    "\n",
    "simple_rnn_history = simple_rnn_model.fit(\n",
    "    post_padded_X,\n",
    "    Y,\n",
    "    epochs=epochs,\n",
    "    batch_size=batch_size,\n",
    "    validation_split=0.1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the training and validation set accuracies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "History: [0.5032, 0.4916, 0.5076]\n",
      "Max accuracy: 0.5076\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXxU9b3/8dcnO0tYQrAoiGAvCgESlhRQFERkURHqUgT1Vr1VrK31qr209uqvWmx7vdVaba+XVlGvPh4ColwtrqgFf2r9oQRlEVQ2UQIUIYSwZpnk+/vjnEwmYZIMMMkkh/fz8chj5qzzmcPwPme+3zPnmHMOEREJrqREFyAiIk1LQS8iEnAKehGRgFPQi4gEnIJeRCTgUhJdQF3Z2dmuV69eiS5DRKRVWbFixW7nXNdo01pc0Pfq1YuCgoJElyEi0qqY2Vf1TVPTjYhIwCnoRUQCTkEvIhJwLa6NXqS1q6iooLCwkNLS0kSXIgGUkZFBjx49SE1NjXkZBb1InBUWFpKZmUmvXr0ws0SXIwHinKOoqIjCwkJ69+4d83JquhGJs9LSUrp06aKQl7gzM7p06XLU3xYV9CJNQCEvTeVYPlsKehGRgFPQiwTMmDFjWLx4ca1xDz/8MDfffHODy7Vv3x6A7du3c8UVV0Sd57zzzmv0B40PP/wwhw4dCg9fdNFF7N27N5bS42bLli3MnTu3WV+zJVPQiwTM9OnTmT9/fq1x8+fPZ/r06TEtf8opp/DCCy8c8+vXDfrXXnuNTp06HfP6jkVLCfpQKJToEgAFvUjgXHHFFbz66quUl5cDXuht376dc889lwMHDjB27FiGDBnCwIED+etf/3rE8lu2bGHAgAEAHD58mGnTptGvXz8uvfRSDh8+HJ7v5ptvJj8/n/79+3PPPfcA8Mc//pHt27czZswYxowZA3iXNdm9ezcADz30EAMGDGDAgAE8/PDD4dfr168fN954I/3792f8+PG1Xqfa888/z4ABA8jLy2PUqFEAVFZWMnPmTL7zne+Qm5vLX/7yFwDuvPNO3nvvPQYNGsQf/vCHWutpaBs888wz5ObmkpeXxz//8z8DsHPnTi699FLy8vLIy8vjgw8+qLWNAB588EHuvfdewPvWc9ttt5Gfn88jjzzCyy+/zPDhwxk8eDAXXHABO3fuDNdx/fXXM3DgQHJzc1m4cCFPPvkkt912W3i9jz/+OLfffnsD/9qx0emVIk3oVy+vZd32fXFdZ84pHbjnkv71Ts/KymLYsGG8/vrrTJkyhfnz5zN16lTMjIyMDF588UU6dOjA7t27GTFiBJMnT663g2/27Nm0bduWzz77jNWrVzNkyJDwtN/85jdkZWVRWVnJ2LFjWb16NbfeeisPPfQQS5cuJTs7u9a6VqxYwVNPPcWHH36Ic47hw4czevRoOnfuzIYNG5g3bx6PP/44U6dOZeHChVxzzTW1lp81axaLFy+me/fu4aagJ554go4dO7J8+XLKysoYOXIk48eP5/777+fBBx/klVdeOeI91bcN1q1bx69//Ws++OADsrOz2bNnDwC33noro0eP5sUXX6SyspIDBw5QXFzc4L9ReXl5uImruLiYZcuWYWbMmTOH3/3ud/z+97/nvvvuo2PHjqxZsyY8X2pqKr/5zW944IEHSE1N5amnngrvvI6Hgl4kgKqbb6qD/oknngC887D//d//nXfffZekpCS2bdvGzp076datW9T1vPvuu9x6660A5ObmkpubG562YMECHnvsMUKhEDt27GDdunW1ptf1/vvvc+mll9KuXTsALrvsMt577z0mT55M7969GTRoEABDhw5ly5YtRyw/cuRIrrvuOqZOncpll10GwJtvvsnq1avDTU0lJSVs2LCBtLS0euuobxssWbKE733ve+EdVFZWFgBLlizhmWeeASA5OZmOHTs2GvRXXnll+HlhYSFXXnklO3bsoLy8PHz++9tvv12ria1z584AnH/++bzyyiv069ePiooKBg4c2OBrxUJBL9KEGjrybkpTpkzh9ttv5+OPP+bQoUMMHToUgGeffZZdu3axYsUKUlNT6dWr1zH9gvfLL7/kwQcfZPny5XTu3JnrrrvuuH4JnJ6eHn6enJwctenmz3/+Mx9++CGvvvoqQ4cOZcWKFTjn+NOf/sSECRNqzfvOO+/U+1rx2AYpKSlUVVWFh+suX70zA/jJT37CHXfcweTJk3nnnXfCTTz1ueGGG/jtb39L3759uf7664+qrvqojV4kgNq3b8+YMWP4l3/5l1qdsCUlJZx00kmkpqaydOlSvvqq3ivbAjBq1Khwp+ann37K6tWrAdi3bx/t2rWjY8eO7Ny5k9dffz28TGZmJvv37z9iXeeeey4vvfQShw4d4uDBg7z44ouce+65Mb+nTZs2MXz4cGbNmkXXrl3ZunUrEyZMYPbs2VRUVACwfv16Dh48WG8NDW2D888/n+eff56ioiKAcNPN2LFjmT17NuD1CZSUlPCtb32Lb775hqKiIsrKyqI2EUW+Xvfu3QF4+umnw+PHjRvHo48+Gh6u/pYwfPhwtm7dyty5c2PuQG+Mgl4koKZPn86qVatqhcXVV19NQUEBAwcO5JlnnqFv374NruPmm2/mwIED9OvXj1/+8pfhbwZ5eXkMHjyYvn37ctVVVzFy5MjwMjNmzGDixInhzthqQ4YM4brrrmPYsGEMHz6cG264gcGDB8f8fmbOnMnAgQMZMGAAZ599Nnl5edxwww3k5OQwZMgQBgwYwE033UQoFCI3N5fk5GTy8vKO6Iytbxv079+fu+66i9GjR5OXl8cdd9wBwCOPPMLSpUsZOHAgQ4cOZd26daSmpvLLX/6SYcOGMW7cuAa347333sv3vvc9hg4dWqvf4u6776a4uDjcwbx06dLwtKlTpzJy5Mhwc87xMudcXFYUL/n5+U43HpHW7LPPPqNfv36JLkNasUmTJnH77bczduzYqNOjfcbMbIVzLj/a/DqiFxFpIfbu3csZZ5xBmzZt6g35Y6HOWBGRFqJTp06sX78+7uvVEb2ISMDFFPRmNtHMvjCzjWZ2Zz3zTDWzdWa21szmRoyvNLOV/t+ieBUuIiKxabTpxsySgUeBcUAhsNzMFjnn1kXM0wf4BTDSOVdsZidFrOKwc25QnOsWEZEYxXJEPwzY6Jzb7JwrB+YDU+rMcyPwqHOuGMA59018yxQRkWMVS9B3B7ZGDBf64yKdAZxhZn83s2VmNjFiWoaZFfjjv3uc9YpII4qKihg0aBCDBg2iW7dudO/ePTxcfaGzxlx//fV88cUXDc7z6KOP8uyzz8aj5KOyZMkSli1b1uyv25rF66ybFKAPcB7QA3jXzAY65/YCpznntpnZ6cASM1vjnNsUubCZzQBmAPTs2TNOJYmcmLp06cLKlSsB78c67du359/+7d9qzeOcwzlHUlL0Y72nnnqq0df58Y9/fPzFHoMlS5aQnZ3NiBEjEvL61SorK0lOTk5oDbGK5Yh+G3BqxHAPf1ykQmCRc67COfclsB4v+HHObfMfNwPvAEf8FM4595hzLt85l9+1a9ejfhMi0riNGzeSk5PD1VdfTf/+/dmxYwczZswIX2p41qxZ4XnPOeccVq5cSSgUolOnTtx5553k5eVx1lln8c03Xsvs3XffHb7U8DnnnMOdd97JsGHDOPPMM/nggw8AOHjwIJdffjk5OTlcccUV5Ofnh3dCkWbOnElOTg65ubn8/Oc/B7zLA1922WXk5+czbNgwli1bxqZNm5gzZw4PPPAAgwYNCr9OtWXLlnHWWWcxePBgRo4cyYYNGwDvuvC33347AwYMIDc3l//+7/8G4MMPP+Sss84iLy+P4cOHc+jQIebMmVPrUsETJ07k/fffD2+L2267jdzcXD766CPuuecevvOd7zBgwAB++MMfUv0D1PXr13P++eeTl5fHkCFD2LJlC1dddVWtSyVceeWVvPrqq8f3jxqjWI7olwN9zKw3XsBPA66qM89LwHTgKTPLxmvK2WxmnYFDzrkyf/xI4Hdxq16kpXv9TvjHmvius9tAuPD+Y1r0888/55lnniE/3/sB5f33309WVhahUIgxY8ZwxRVXkJOTU2uZkpISRo8ezf33388dd9zBk08+yZ13HnnynXOOjz76iEWLFjFr1izeeOMN/vSnP9GtWzcWLlzIqlWral3muNrOnTt57bXXWLt2LWYWvgTxrbfeys9+9jNGjBjBli1bmDRpEp9++ik33HAD2dnZtcK4Wr9+/XjvvfdISUnhjTfe4O677+a5555j9uzZbN++nVWrVpGcnMyePXsoLS1l2rRpLFy4kCFDhlBSUlLr4mrRlJSUMGrUqPAO7swzz+RXv/oVzjmuuuoq3njjDS688EKmT5/OvffeyyWXXEJpaSlVVVX84Ac/YPbs2UyaNIni4mKWL1/ebDdHaTTonXMhM7sFWAwkA08659aa2SygwDm3yJ823szWAZXATOdckZmdDfzFzKrwvj3cH3m2jog0r29/+9vhkAeYN28eTzzxBKFQiO3bt7Nu3bojgr5NmzZceOGFgHcJ4ffeey/quqsvHRx5meH3338/fISel5dH//5HXs0zKyuLpKQkbrzxRi6++GImTZoEeJfxjewnKC4ujnpVy0h79+7l+9//Pps21Wod5u233+a2224LN7VkZWXxySef0LNnz/DOp2PHjg2uGyAtLY1LL700PPy3v/2NBx54gNLSUnbv3s3QoUMZMWIEu3fv5pJLLgG869+Dd9G0W265haKiIubNm8fUqVObreknpjZ659xrwGt1xv0y4rkD7vD/Iuf5ADj+iymLtFbHeOTdVCIvn7thwwYeeeQRPvroIzp16sQ111wT9XK9kdd2T05Orvf2eNVHww3NE01qaioFBQW89dZbPP/888yePZs333wz/A2hoWvL13XXXXcxYcIEfvSjH7Fx40YmTpzY+EJ1NHQJ4jZt2oRv0nLo0CFuueUWPv74Y7p3787dd9/d4OWOzYxrrrmGuXPn8vTTTzdrR7Z+GStygtq3bx+ZmZl06NCBHTt2HHFD8XgYOXIkCxYsAGDNmjWsW3fkF/r9+/ezb98+Jk2axB/+8Ac++eQTAC644IJal/Gtbttv7BLE1ZcE/p//+Z/w+HHjxvHnP/+ZyspKwLsEcU5ODl9//TUff/wx4G2PyspKevXqxSeffIJzji1btrBixYqor3X48GGSkpLIzs5m//79LFy4EPBuINK1a1defvllwNtRVN9D9/rrr+eBBx4gPT2dM888M4YtGB8KepET1JAhQ8jJyaFv3758//vfr3Wp4Xj5yU9+wrZt28jJyeFXv/oVOTk5RzSRlJSUcPHFF5OXl8fo0aN56KGHAO/0zb///e/k5uaSk5PD448/Dng3VVmwYAGDBw8+ojP25z//OTNnzmTIkCHhjlGAm266iW7duoXvB7tgwQLS09OZN28eN998M3l5eYwfP56ysjJGjx5N9+7d6devHz/96U/Dd76qq0uXLlx77bXk5ORw4YUXMnz48PC0Z599lt///vfk5uZyzjnnsGvXLsC78foZZ5wRtxuKxEqXKRaJM12muEYoFCIUCpGRkcGGDRsYP348GzZsICXlxLye4sGDBxk4cCCrVq0iMzPzmNdztJcpPjG3tog0iwMHDjB27FhCoRDOOf7yl7+csCG/ePFibrzxRmbOnHlcIX8sTswtLiLNolOnTvW2cZ9oJkyYwNdff52Q11YbvUgTaGlNohIcx/LZUtCLxFlGRgZFRUUKe4k75xxFRUXhc/NjpaYbkTjr0aMHhYWF4TMtROIpIyODHj16HNUyCnqROEtNTaV3796JLkMkTE03IiIBp6AXEQk4Bb2ISMAp6EVEAk5BLyIScAp6EZGAU9CLiAScgl5EJOAU9CIiAaegFxEJOAW9iEjAKehFRAJOQS8iEnAKehGRgFPQi4gEnIJeRCTgFPQiIgGnoBcRCTgFvYhIwCnoRUQCLqagN7OJZvaFmW00szvrmWeqma0zs7VmNjdi/LVmtsH/uzZehYuISGxSGpvBzJKBR4FxQCGw3MwWOefWRczTB/gFMNI5V2xmJ/njs4B7gHzAASv8ZYvj/1ZERCSaWI7ohwEbnXObnXPlwHxgSp15bgQerQ5w59w3/vgJwFvOuT3+tLeAifEpXUREYhFL0HcHtkYMF/rjIp0BnGFmfzezZWY28SiWxcxmmFmBmRXs2rUr9upFRKRR8eqMTQH6AOcB04HHzaxTrAs75x5zzuU75/K7du0ap5JERARiC/ptwKkRwz38cZEKgUXOuQrn3JfAerzgj2VZERFpQrEE/XKgj5n1NrM0YBqwqM48L+EdzWNm2XhNOZuBxcB4M+tsZp2B8f44ERFpJo2edeOcC5nZLXgBnQw86Zxba2azgALn3CJqAn0dUAnMdM4VAZjZfXg7C4BZzrk9TfFGREQkOnPOJbqGWvLz811BQUGiyxARaVXMbIVzLj/aNP0yVkQk4BT0IiIBp6AXEQk4Bb2ISMAp6EVEWoomOjmm0dMrRUQCqaoSQmVQWQahcu+xsuLIceHHMqgs9/6qn0d9rF6mPMq4KI+V5TXPTxkCN7wV97eqoBeRplcZOjLU6gZo3cd6A7SekGxwfRVHTnOV8Xt/SSmQnA4pad5jclrN88jH9ExI8afXmscf16ln/GqKoKAXCRLnvCPVegPvWI5eGwnkWILZVcXvPSal+sGYWidIqwM2HVIyIKNjRKCmR3mMDObUI8dFW29960tKjt/7awIKepFj5RxUhRoI0ngfvTYUzBHrJ47tvOFQbSAkU9tAm07Rp9UKywaCOXLZ5LqvWedIOUldi0dLQS/SmAO7YM0CWL0ASgprNwPEM1STo3zVDz/6z9PaQnLnozvajBq+9b1WnWXN4vf+JGEU9CLRhMph/Ruwah5seNM7cj9lMORMrhOoDYRltDbYho5eFarSRBT0ItWcgx2rYOVcWPM8HN4D7bvBWT+GvKvgpL6JrlDkmCjoRfbv9JpmVs6Fb9Z5R9p9L4JBV8PpYyBZ/02kddMnWE5MoTL44nUv3De+7Z1q1z0fLn4IBlwGbTonukKRuFHQy4nDOdj+Mayc5zXNlO6FzJNh5K1e00zXMxJdoUiTUNBL8O3/B6x+zjt63/W5d45130kwaLrXNNPCz4EWOV4KegmmilL44jUv3Df9zfvBTo9hMOlh6H+pd963yAlCQS/B4RxsWwErn4VPF0JpCXToDufcDnnTIbtPoisUSQgFvbR++7bDqvneOe+710NKG+h3CQy6CnqPUtOMnPAU9NI6VRyGz1/1mmY2L/WaZnqeBZf8Efp/17vOiYgACnppTZyDwuV+08yLUFYCHXrAuT/1mma6fDvRFYq0SAp6aflKCmuaZoo2ek0zOVO8pple5+oiVyKNUNBLy1R+yG+aeRY2vwM4OG0kjLzNC/mMDomuUKTVUNBLy+EcbP2wpmmmfL93I4bRP4O8aZB1eqIrFGmVFPSSeHu3+k0zc2HPZkhtV9M0c9pINc2IHCcFvSRG+UH47GXvrJkv3wWc194+aib0mwzp7RNdoUhgKOil+TgHX33gHbmvfQnKD0Cn0+C8X0DeldC5V6IrFAkkBb00veKvappmirdAWnvI+a7XNNPzLDXNiDSxmILezCYCjwDJwBzn3P11pl8HPABs80f9l3Nujj+tEljjj//aOTc5DnVLS1d2AD5b5DXNbHnPG9d7lHf03u8SSGuX2PpETiCNBr2ZJQOPAuOAQmC5mS1yzq2rM+tzzrlboqzisHNu0PGXKi1eVRV8/YEX7mtfgoqD0Lk3jLnba5rp1DPRFYqckGI5oh8GbHTObQYws/nAFKBu0MuJas+XNU0ze7+GtEzv5h2DroaeI3QvVJEEiyXouwNbI4YLgeFR5rvczEYB64HbnXPVy2SYWQEQAu53zr1Ud0EzmwHMAOjZU0d9rULZflj3V+/o/au/Awanj4bz/493rfe0tomuUER88eqMfRmY55wrM7ObgKeB8/1ppznntpnZ6cASM1vjnNsUubBz7jHgMYD8/HwXp5ok3qqqvPb2lXO99veKQ5D1bS/c86ZBxx6JrlBEoogl6LcBp0YM96Cm0xUA51xRxOAc4HcR07b5j5vN7B1gMFAr6KWFK9pUc62Zkq2Q3gFyp3q33zt1mJpmRFq4WIJ+OdDHzHrjBfw04KrIGczsZOfcDn9wMvCZP74zcMg/0s8GRhKxE5AWrHQfrHvJO3r/+v8BBt8+Hy64F/peDKltElygiMSq0aB3zoXM7BZgMd7plU8659aa2SygwDm3CLjVzCbjtcPvAa7zF+8H/MXMqoAkvDZ6deK2VFWV3q9UV871frUaOgxd+sDYeyD3SujYPdEVisgxMOdaVpN4fn6+KygoSHQZJ5bdG70zZlY9B/sKIb0jDLzcO2um+1A1zYi0Ama2wjmXH22afhl7oiotgbUvekfvWz8ES4Jvj4Xx98GZF0FqRqIrFJE4UdCfSKoqvWu7r5wLn78CoVLIPhMu+JXXNNPh5ERXKCJNQEF/Iti1vqZpZv92yOgEg6/xrjVzyhA1zYgEnII+qA4Xw6f/650SWbgcLBn+6QKY+B9w5oWQkp7oCkWkmSjog6SqEjYt9e7Q9PmrUFkGXfvBuPu8894zuyW6QhFJAAV9EHzzeU3TzIF/QJvOMPRar2nm5EFqmhE5wSnoW6tDe+DThV7TzLYVXtNMn/FeuJ8xQU0zIhKmoG9NKkOw6W/eWTNfvAaV5XBSf5jwWxj4PWh/UqIrFJEWSEHfGuxc5zXNrF4AB3ZC2y6Q/wMYNB265appRkQapKBvqQ7tgTUveB2rO1ZCUgr0meA1zfQZDylpia5QRFoJBX1LUlkBG9/2m2Zeh6oK6DYQJt7vNc20y050hSLSCinoW4J/fOp1qq5+Dg7ugrbZMGyG3zQzMNHViUgrp6BPlINFsOZ5r2nmH6shKRXOnOhd473POEhOTXSFIhIQCvrmVFkBG970mmbWL/aaZk7Ogwt/BwOugHZdEl2hiASQgr457FjtN80sgEO7od1JMPwmr2P1W/0TXZ2IBJyCvqkc2OU3zcyFnWsgOc27xkzeVfBPY9U0IyLNRkEfT6Fy2LDYC/cNb0JVyLs65EUPwoDLoW1WoisUkROQgv54OQc7VnnhvuZ5OLwH2n8LRvzIa5o5qV+iKxSRE5yC/lgd+MZrc185F75Z6zXN9L3Yu/3e6WMgWZtWRFoGpdHRCJXB+jf8ppm3wFVC93y4+CEYcJl31UgRkRZGQd8Y52D7J164f/qCd0OPzJPh7J94TTNdz0x0hSIiDVLQ12f/P7xfqq6cB7s+g+R06DfJC/fTx0BScqIrFBGJiYI+UkUprH/dO3rf+Da4KugxDCY9DP0vhTadEl2hiMhRU9A7B9s+9i5F8OkLUFoCHbrDObdD3nTI7pPoCkVEjsuJG/T7ttc0zez+AlIyoN8lXtNM79FqmhGRJuecY39ZiOKD5ew5WE6SGXmnxr/l4MQK+orD3k2zV82DTUu8pplTR8Alf4T+34WMjomuUERaKecchysq2XOwnOKDFew5VB4O8OJDdR4jpoeqXHgdead24q8/Hhn32oIf9M5BYYHfNPO/UFYCHXrAuT/1mma6fDvRFYpIC1QWqmTvoQo/mMsjgruiVnAXHagJ8LJQVdR1JRl0bptG53ZpZLVNo1d2W4a060TntmlktUsLP36rQ0aTvJfABL1z3l7Rqm+rV7INVs/3OlaLNkJKG8iZ4l3jvdcoSEpKYLUi0pxClVXsPVxR5wg7IrAjg/yQd8R9oCxU7/o6ZKR4Ad0ujZM7ZpBzSoeIwE6tCXA/2Du2SSUpKXG3/Iwp6M1sIvAIkAzMcc7dX2f6dcADwDZ/1H855+b4064F7vbH/9o593Qc6j5CyeEKzr7vFSanf8JlSf+X/KrVJOH4In0gBd1+xsbssaRldCRzSwrtd3xFZkYq7TNSyMxIITM9lcyMlPBweora50Vaqqoqx77SitqBXeuIO7KpxJuv5HBFvetrl5bsBbIf1Kd3bV8T2H5QR07v1DaV1OTWdaDYaNCbWTLwKDAOKASWm9ki59y6OrM+55y7pc6yWcA9QD7ggBX+ssVxqT5CSskWPml7C+mVB9mT3I3XOl7D22nns7nyJPYfCLF/9372lxbX+9UqUlpykrcDqA7/9MidQkqtnUT79BQ61BnOzEilfXoKyQncg4u0Bs45DpZXhgP6yLCOHuIRzdq1pKUk0SWiKaR757ZktU2tFdSRj53appKRGvwDu1iO6IcBG51zmwHMbD4wBagb9NFMAN5yzu3xl30LmAjMO7Zy69e+Wx8Ydj2cMYGs085hUlISk6LMVx6q4kBZiP2lFewvDbG/NBQe9h6r/2qGD5SG2LrnUMT0ino/aJHapSX7O4BUfwcQ7RtEqr/zSKk1bwd/uE1qck1zlEgLV+p3RtbueCxnz6HogV18sILyyugHX8lJVqsp5J9Oak/ndmm1grzmiDuVrHZp+v9Sj1iCvjuwNWK4EBgeZb7LzWwUsB643Tm3tZ5lu9dd0MxmADMAevbsGVvlR64EJvym0dnSUpLISvE+JMequne9vp3CvjrD+8tqdio7Skq9+UtDHCyvbPS1kpMsvJOo79tDZpThuvO2tq+akngVlVXhMC46WNbgmSTFB70mksMV0T/TZtCpTU1TyKlZbcnr0ck/0j6yTbtzuzQ6ZKQotOMkXp2xLwPznHNlZnYT8DRwfqwLO+ceAx4DyM/Pj+FYObHMjLZpKbRNS+FbHY59PZVV7ohvE9U7ilrfNKp3KP7wN/tL2bQrFB5f3xFRpPSUpCg7hRTap9eMixxun+F/q4gYbp+WktAOJTl2lVWOksMVUY60o59JsudgOftL6++MzKzujGybxkmZGZz5rQ71tmlntfM6I9WUmTixBP024NSI4R7UdLoC4JwrihicA/wuYtnz6iz7ztEWGVTJSUbHNql0bHN8d5sqC1XWfHuI+AZxoM63jX11dh5bdnvNUdXfQFwMu9jIbxeZGTX9FR3q9E/U7Dxqhr35UklPSdKR2nFwzrGvNFRPB2RFlDNIyjtUb6oAAAt3SURBVNl7uKLef9+2acm1jqh7dWlbb2B3bpdKpzZppKXoG2JrEkvQLwf6mFlvvOCeBlwVOYOZneyc2+EPTgY+858vBn5rZtXX7x0P/OK4q5Za0lOSSW+fTHb79GNeR3WnWPXOYX9k05O/I9gXZeex93AFW4sPheet76t7pJQkO6Kju9aOImInkZlefzNVSgCaoxr7kU3RwSODfO+h2j+yiZSWnETniKaQfid3qAnsejol26QFvzPyRNdo0DvnQmZ2C15oJwNPOufWmtksoMA5twi41cwmAyFgD3Cdv+weM7sPb2cBMKu6Y1ZaFjOvL6B9egrdOh77jzYqKqs4WE+n9v46TVGRneLb95bWGq4vyCK1SU0+8myoOh3b9XV0V+9Q2qXFt/OuLFQZbq8+8teQEZ2SEePrOxPM64xMDf/Q5vTs9gw9LXqbdvXzeL8fCQZzsXxfb0b5+fmuoKAg0WVIAjnnKAtVhXcUdXcKdc+UivZN44C/Y2mMGTUd1/Wd/RQen+oHefQ27eKD5Q12sHdqmxo+uo56nnZEU0lW2zQyM9QnIrEzsxXOufxo0wLzy1gJDjMjIzWZjNRkumYee3NUVZXjYPmR3x5qdhbRh/ccLOerokPhHU20I+726SneKX3+0fQ/dW1fE9IRTSPVR98d26QGoqlJWicFvQRWUpL57frH19ld/duLA6Uh0lOT6NQ2Vb+ellZFQS/SiHj89kIkkfRdUkQk4BT0IiIBp6AXEQk4Bb2ISMAp6EVEAk5BLyIScAp6EZGAU9CLiAScgl5EJOAU9CIiAaegFxEJOAW9iEjAKehFRAJOQS8iEnAKehGRgFPQi4gEnIJeRCTgFPQiIgGnoBcRCTgFvYhIwCnoRUQCTkEvIhJwCnoRkYBT0IuIBJyCXkQk4GIKejObaGZfmNlGM7uzgfkuNzNnZvn+cC8zO2xmK/2/P8ercBERiU1KYzOYWTLwKDAOKASWm9ki59y6OvNlAv8KfFhnFZucc4PiVK+IiBylWI7ohwEbnXObnXPlwHxgSpT57gP+EyiNY30iInKcYgn67sDWiOFCf1yYmQ0BTnXOvRpl+d5m9omZ/V8zOzfaC5jZDDMrMLOCXbt2xVq7iIjE4Lg7Y80sCXgI+GmUyTuAns65wcAdwFwz61B3JufcY865fOdcfteuXY+3JBERiRBL0G8DTo0Y7uGPq5YJDADeMbMtwAhgkZnlO+fKnHNFAM65FcAm4Ix4FC4iIrGJJeiXA33MrLeZpQHTgEXVE51zJc65bOdcL+dcL2AZMNk5V2BmXf3OXMzsdKAPsDnu70JEROrV6Fk3zrmQmd0CLAaSgSedc2vNbBZQ4Jxb1MDio4BZZlYBVAE/dM7tiUfhIiISG3POJbqGWvLz811BQUGiyxARaVXMbIVzLj/aNP0yVkQk4BT0IiIBp6AXEQk4Bb2ISMAp6EVEAk5BLyIScAp6EZGAU9CLiAScgl5EJOAU9CIiAaegFxEJOAW9iEjAKehFRAJOQS8iEnAKehGRgFPQi4gEnIJeRCTgFPQiIgGnoBcRCTgFvYhIwCnoRUQCTkEvIhJwCnoRkYBT0IuIBJyCXkQk4BT0IiIBp6AXEQk4Bb2ISMDFFPRmNtHMvjCzjWZ2ZwPzXW5mzszyI8b9wl/uCzObEI+iRUQkdimNzWBmycCjwDigEFhuZoucc+vqzJcJ/CvwYcS4HGAa0B84BXjbzM5wzlXG7y2IiEhDYjmiHwZsdM5tds6VA/OBKVHmuw/4T6A0YtwUYL5zrsw59yWw0V+fiIg0k1iCvjuwNWK40B8XZmZDgFOdc68e7bIiItK0jrsz1sySgIeAnx7HOmaYWYGZFezatet4SxIRkQixBP024NSI4R7+uGqZwADgHTPbAowAFvkdso0tC4Bz7jHnXL5zLr9r165H9w5ERKRBsQT9cqCPmfU2szS8ztVF1ROdcyXOuWznXC/nXC9gGTDZOVfgzzfNzNLNrDfQB/go7u9CRETq1ehZN865kJndAiwGkoEnnXNrzWwWUOCcW9TAsmvNbAGwDggBP9YZNyIizcucc4muoZb8/HxXUFCQ6DJERFoVM1vhnMuPNk2/jBURCbgWd0RvZruAr45jFdnA7jiVE0+q6+iorqOjuo5OEOs6zTkX9WyWFhf0x8vMCur7+pJIquvoqK6jo7qOzolWl5puREQCTkEvIhJwQQz6xxJdQD1U19FRXUdHdR2dE6quwLXRi4hIbUE8ohcRkQgKehGRgGs1Qd/YXa786+k850//0Mx6RUxrsrtcxVDXHWa2zsxWm9nfzOy0iGmVZrbS/6v3UhJNVNd1ZrYr4vVviJh2rZlt8P+ubea6/hBR03oz2xsxrSm315Nm9o2ZfVrPdDOzP/p1r/YvzV09rSm3V2N1Xe3Xs8bMPjCzvIhpW/zxK80srj83j6Gu88ysJOLf65cR02K6Y10T1TUzoqZP/c9Ulj+tKbfXqWa21M+CtWb2r1HmabrPmHOuxf/hXWNnE3A6kAasAnLqzPMj4M/+82nAc/7zHH/+dKC3v57kZqxrDNDWf35zdV3+8IEEbq/rgP+KsmwWsNl/7Ow/79xcddWZ/yd411Zq0u3lr3sUMAT4tJ7pFwGvA4Z3hdYPm3p7xVjX2dWvB1xYXZc/vAXITtD2Og945Xg/A/Guq868lwBLmml7nQwM8Z9nAuuj/J9sss9Yazmij+UuV1OAp/3nLwBjzcxo2rtcNVqXc26pc+6QP7gM71LNTS3Wu4JFMwF4yzm3xzlXDLwFTExQXdOBeXF67QY5594F9jQwyxTgGedZBnQys5Np2u3VaF3OuQ/814Xm+3zFsr3qczyfzXjX1Zyfrx3OuY/95/uBzzjyJkxN9hlrLUEfy52qwvM450JACdAlxmWbsq5IP8DbY1fLMO+GK8vM7Ltxqulo6rrc/4r4gplV3zegRWwvv4mrN7AkYnRTba9Y1Fd7S7qLWt3PlwPeNLMVZjYjAfWcZWarzOx1M+vvj2sR28vM2uKF5cKI0c2yvcxrVh5MxP21fU32GWv0MsUSH2Z2DZAPjI4YfZpzbpuZnQ4sMbM1zrlNzVTSy8A851yZmd2E923o/GZ67VhMA15wtS9rncjt1aKZ2Ri8oD8nYvQ5/vY6CXjLzD73j3ibw8d4/14HzOwi4CW8+1G0FJcAf3fORR79N/n2MrP2eDuX25xz++K57oa0liP6WO5UFZ7HzFKAjkBRjMs2ZV2Y2QXAXXg3ZCmrHu+c2+Y/bgbewdvLN0tdzrmiiFrmAENjXbYp64owjTpfq5twe8WivtqbcnvFxMxy8f4NpzjniqrHR2yvb4AXiV+TZaOcc/uccwf8568BqWaWTQvYXr6GPl9Nsr3MLBUv5J91zv1vlFma7jPWFB0P8f7D++axGe+rfHUHTv868/yY2p2xC/zn/andGbuZ+HXGxlLXYLzOpz51xncG0v3n2cAG4tQpFWNdJ0c8vxRY5mo6fr706+vsP89qrrr8+fridYxZc2yviNfoRf2dixdTu6Pso6beXjHW1ROv3+nsOuPbAZkRzz8AJjZjXd2q//3wAvNrf9vF9Bloqrr86R3x2vHbNdf28t/7M8DDDczTZJ+xuG3cpv7D65Fejxead/njZuEdJQNkAM/7H/qPgNMjlr3LX+4L4MJmruttYCew0v9b5I8/G1jjf9DXAD9o5rr+A1jrv/5SoG/Esv/ib8eNwPXNWZc/fC9wf53lmnp7zQN2ABV4baA/AH4I/NCfbsCjft1rgPxm2l6N1TUHKI74fBX440/3t9Uq/9/5rmau65aIz9cyInZE0T4DzVWXP891eCdoRC7X1NvrHLw+gNUR/1YXNddnTJdAEBEJuNbSRi8iIsdIQS8iEnAKehGRgFPQi4gEnIJeRCTgFPQiIgGnoBcRCbj/D2gopAIITQeCAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "def plot_history(history):\n",
    "    train_metric = 'acc' if 'acc' in history.history else 'accuracy'\n",
    "    val_metric = 'val_acc' if 'val_acc' in history.history else 'val_accuracy'\n",
    "    print(\"History:\", history.history[val_metric])\n",
    "    print(\"Max accuracy:\", max(history.history[val_metric]))\n",
    "    y_min = min(history.history[train_metric] + history.history[val_metric])\n",
    "    y_max = max(history.history[train_metric] + history.history[val_metric])\n",
    "    plt.ylim(max(y_min-0.1, 0.0), min(y_max+0.1, 1.0))\n",
    "    plt.plot(history.history[val_metric],label=\"Validation set accuracy\")\n",
    "    plt.plot(history.history[train_metric],label=\"Training set accuracy\")\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "plot_history(simple_rnn_history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, that's not learning much of anything.\n",
    "\n",
    "(What's going on here?)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Post-padded:\n",
      "[    9   215 20597  1719    20     5   192   592    14     1    82    54\n",
      "     2     9    24     5   131     9    62   254     8     5    26   175\n",
      "    48    43    21    22     3   333     4  7261 13419    21    76   115\n",
      "     8     1  2931    16   300   269  1000    23   326    38   123     8\n",
      "     1   112    35   312   330     6     3   113  3308     2  5073     2\n",
      "    45     4     1   135    96    26   601   176    17     8   491    42\n",
      "     7     1   126     1  2252    19     1   284    22    39    13    83\n",
      "    13     1    18    43    20  2780    14   144  8057   447     8    44\n",
      "  3395     4 14895   384    35     1   266 13420     2     3    83  2695\n",
      "    15     1   163   255    33    44   820   139   227    31  1720    18\n",
      "    76   115     5   102  8058     1   163  8057     2   312   330    19\n",
      "     1    51   388   487   961   266   349     5  4009    42    10   113\n",
      " 12257   943    18     8    12   180   733     2   142  1259     5    24\n",
      "     8     7    57  1587     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0]\n"
     ]
    }
   ],
   "source": [
    "print('Post-padded:')\n",
    "print(post_padded_X[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`SimpleRNN` implements the basic RNN cell structure introduced as \"vanilla RNN\" on the lectures. What we're seeing here are the training difficulties we discussed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train LSTM\n",
    "\n",
    "Let's try the exact same process, except with `LSTM` instead of `SimpleRNN`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/smp/Library/Python/3.7/lib/python/site-packages/tensorflow_core/python/ops/math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
      "Train on 22500 samples, validate on 2500 samples\n",
      "Epoch 1/3\n",
      "22500/22500 [==============================] - 1166s 52ms/sample - loss: 0.6809 - acc: 0.5485 - val_loss: 0.6594 - val_acc: 0.5840\n",
      "Epoch 2/3\n",
      "22500/22500 [==============================] - 1218s 54ms/sample - loss: 0.5343 - acc: 0.7284 - val_loss: 0.4214 - val_acc: 0.8268\n",
      "Epoch 3/3\n",
      "22500/22500 [==============================] - 1119s 50ms/sample - loss: 0.2925 - acc: 0.8892 - val_loss: 0.3678 - val_acc: 0.8584\n"
     ]
    }
   ],
   "source": [
    "lstm_model = build_rnn_model(LSTM, sequence_length, vocab_size, num_classes)\n",
    "\n",
    "lstm_model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "\n",
    "lstm_history = lstm_model.fit(post_padded_X, Y, epochs=epochs, batch_size=batch_size, validation_split=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "History: [0.584, 0.8268, 0.8584]\n",
      "Max accuracy: 0.8584\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3dd3hUVf7H8fchFRJISEINJSAICSkQAqFKkyZdFAFZBQt2f+IuKyprYVeXtYKuIoig7CosiigoggVUECmhJRB6D51UkpAyyfn9cSdxCEmYwEwmM/m+nofHmXvv3PvlZvzkcu495yitNUIIIZxfDUcXIIQQwjYk0IUQwkVIoAshhIuQQBdCCBchgS6EEC7C3VEHDgoK0iEhIY46vBBCOKVt27Zd1FrXK22dwwI9JCSEuLg4Rx1eCCGcklLqeFnrpMlFCCFchAS6EEK4CAl0IYRwEQ5rQxfC2eXn55OUlEROTo6jSxEuyNvbmyZNmuDh4WH1ZyTQhbhOSUlJ1K5dm5CQEJRSji5HuBCtNcnJySQlJdGiRQurPydNLkJcp5ycHAIDAyXMhc0ppQgMDKzwv/4k0IW4ARLmwl6u57slgS6EEC5CAl0IJ9WnTx/WrFlzxbJZs2bxyCOPlPs5X19fAE6fPs0dd9xR6ja9e/e+Zse/WbNmkZ2dXfz+tttuIy0tzZrSbebYsWN89tlnlXrMqkwCXQgnNW7cOJYsWXLFsiVLljBu3DirPt+4cWO++OKL6z5+yUBftWoV/v7+172/61FVAt1kMjm6BEACXQindccdd/Dtt9+Sl5cHGOF2+vRpevbsSWZmJv369SM6OpqIiAi+/vrrqz5/7NgxwsPDAbh8+TJjx44lNDSUUaNGcfny5eLtHnnkEWJiYmjXrh0vvvgiAO+88w6nT5+mT58+9OnTBzCG87h48SIAb731FuHh4YSHhzNr1qzi44WGhvLggw/Srl07BgwYcMVxinz++eeEh4cTFRXFLbfcAkBBQQFTp06lU6dOREZGMnfuXACmTZvG+vXrad++PW+//fYV+ynvHCxatIjIyEiioqL405/+BMC5c+cYNWoUUVFRREVFsXHjxivOEcAbb7zBSy+9BBj/innqqaeIiYlh9uzZrFy5ktjYWDp06MCtt97KuXPniuuYNGkSERERREZGsmzZMhYsWMBTTz1VvN8PP/yQKVOmlPPTto48tiiEDby8cg+JpzNsus+wxnV4cVi7MtcHBATQuXNnvvvuO0aMGMGSJUsYM2YMSim8vb1Zvnw5derU4eLFi3Tp0oXhw4eXeaNtzpw51KpVi7179xIfH090dHTxuldeeYWAgAAKCgro168f8fHxPPnkk7z11lusW7eOoKCgK/a1bds2Fi5cyObNm9FaExsbS69evahbty4HDx5k8eLFfPjhh4wZM4Zly5YxYcKEKz4/Y8YM1qxZQ3BwcHETzkcffYSfnx9bt24lNzeX7t27M2DAAGbOnMkbb7zBN998c9XfqaxzkJiYyD/+8Q82btxIUFAQKSkpADz55JP06tWL5cuXU1BQQGZmJqmpqeX+jPLy8oqbplJTU9m0aRNKKebPn89rr73Gm2++yd///nf8/PxISEgo3s7Dw4NXXnmF119/HQ8PDxYuXFj8S+pGSKAL4cSKml2KAv2jjz4CjOeYn3vuOX799Vdq1KjBqVOnOHfuHA0bNix1P7/++itPPvkkAJGRkURGRhavW7p0KfPmzcNkMnHmzBkSExOvWF/Shg0bGDVqFD4+PgDcfvvtrF+/nuHDh9OiRQvat28PQMeOHTl27NhVn+/evTsTJ05kzJgx3H777QB8//33xMfHFzcRpaenc/DgQTw9Pcuso6xzsHbtWu68887iX0QBAQEArF27lkWLFgHg5uaGn5/fNQP9rrvuKn6dlJTEXXfdxZkzZ8jLyyt+fvzHH3+8ommsbt26APTt25dvvvmG0NBQ8vPziYiIKPdY1pBAF8IGyruStqcRI0YwZcoUtm/fTnZ2Nh07dgTg008/5cKFC2zbtg0PDw9CQkKuq0fr0aNHeeONN9i6dSt169Zl4sSJN9Qz1svLq/i1m5tbqU0uH3zwAZs3b+bbb7+lY8eObNu2Da017777LgMHDrxi259//rnMY9niHLi7u1NYWFj8vuTni35pATzxxBM8/fTTDB8+nJ9//rm4aaYsDzzwAK+++ipt27Zl0qRJFaqrLNKGLoQT8/X1pU+fPtx3331X3AxNT0+nfv36eHh4sG7dOo4fL3PEVQBuueWW4puLu3fvJj4+HoCMjAx8fHzw8/Pj3LlzfPfdd8WfqV27NpcuXbpqXz179uSrr74iOzubrKwsli9fTs+ePa3+Ox0+fJjY2FhmzJhBvXr1OHnyJAMHDmTOnDnk5+cDcODAAbKyssqsobxz0LdvXz7//HOSk5MBiptc+vXrx5w5cwCjzT49PZ0GDRpw/vx5kpOTyc3NLbVpx/J4wcHBAHzyySfFy/v37897771X/L7oqj82NpaTJ0/y2WefWX0j+1ok0IVwcuPGjWPXrl1XhMLdd99NXFwcERERLFq0iLZt25a7j0ceeYTMzExCQ0N54YUXiq/0o6Ki6NChA23btmX8+PF07969+DOTJ09m0KBBxTdFi0RHRzNx4kQ6d+5MbGwsDzzwAB06dLD67zN16lQiIiIIDw+nW7duREVF8cADDxAWFkZ0dDTh4eE89NBDmEwmIiMjcXNzIyoq6qqbomWdg3bt2vH888/Tq1cvoqKiePrppwGYPXs269atIyIigo4dO5KYmIiHhwcvvPACnTt3pn///uWex5deeok777yTjh07XnFfYfr06aSmphbf6F23bl3xujFjxtC9e/fiZpgbpbTWNtlRRcXExGiZ4EI4s7179xIaGuroMoQTGzp0KFOmTKFfv36lri/tO6aU2qa1jilte7lCF0KISpaWlsbNN99MzZo1ywzz6yE3RYUQopL5+/tz4MABm+9XrtCFEMJFSKALIYSLkEAXQggXIYEuhBAuQgJdCCeVnJxM+/btad++PQ0bNiQ4OLj4fdGAXdcyadIk9u/fX+427733Hp9++qktSq6QtWvXsmnTpko/rjOTp1yEcFKBgYHs3LkTMDq1+Pr68pe//OWKbbTWaK2pUaP0a7eFCxde8ziPPfbYjRd7HdauXUtQUBBdunRxyPGLFBQU4Obm5tAarCVX6EK4mEOHDhEWFsbdd99Nu3btOHPmDJMnTy4eAnfGjBnF2/bo0YOdO3diMpnw9/dn2rRpREVF0bVrV86fPw8YPR2LhsDt0aMH06ZNo3PnzrRp04aNGzcCkJWVxejRowkLC+OOO+4gJiam+JeNpalTpxIWFkZkZCTPPPMMYAxbe/vttxMTE0Pnzp3ZtGkThw8fZv78+bz++uu0b9+++DhFNm3aRNeuXenQoQPdu3fn4MGDgDEu+ZQpUwgPDycyMpL3338fgM2bN9O1a1eioqKIjY0lOzub+fPnXzGE7aBBg9iwYUPxuXjqqaeIjIxky5YtvPjii3Tq1Inw8HAefvhhijpkHjhwgL59+xIVFUV0dDTHjh1j/PjxVwwRcNddd/Htt9/e2A/VSnKFLoQtfDcNzibYdp8NI2DwzOv66L59+1i0aBExMUaHwpkzZxIQEIDJZKJPnz7ccccdhIWFXfGZ9PR0evXqxcyZM3n66adZsGAB06ZNu2rfWmu2bNnCihUrmDFjBqtXr+bdd9+lYcOGLFu2jF27dl0x/G6Rc+fOsWrVKvbs2YNSqnho3CeffJK//vWvdOnShWPHjjF06FB2797NAw88QFBQ0BWhWyQ0NJT169fj7u7O6tWrmT59Ov/73/+YM2cOp0+fZteuXbi5uZGSkkJOTg5jx45l2bJlREdHk56efsUgYaVJT0/nlltuKf5F1qZNG15++WW01owfP57Vq1czePBgxo0bx0svvcSwYcPIycmhsLCQ+++/nzlz5jB06FBSU1PZunVrpU3CIYEuhAu66aabisMcYPHixXz00UeYTCZOnz5NYmLiVYFes2ZNBg8eDBhD265fv77UfRcNaWs5/O2GDRuKr7ijoqJo1+7q0ScDAgKoUaMGDz74IEOGDGHo0KGAMbysZTt+ampqqaMwWkpLS+Oee+7h8OHDVyz/8ccfeeqpp4qbSAICAtixYwfNmjUr/iXj5+dX7r4BPD09GTVqVPH7n376iddff52cnBwuXrxIx44d6dKlCxcvXmTYsGGAMf46GIN/Pf744yQnJ7N48WLGjBlTaU02EuhC2MJ1Xknbi+WwrgcPHmT27Nls2bIFf39/JkyYUOowspZji7u5uZU5rVrR1W1525TGw8ODuLg4fvjhBz7//HPmzJnD999/X3zFX97Y5iU9//zzDBw4kEcffZRDhw4xaNAgqz9bpLyhcWvWrFk8GUh2djaPP/4427dvJzg4mOnTp5c7DK9SigkTJvDZZ5/xySefVOoNZWlDF8LFZWRkULt2berUqcOZM2eumljaFrp3787SpUsBSEhIIDEx8aptLl26REZGBkOHDuXtt99mx44dANx6661XDC9b1PZ+raFxi4aq/fjjj4uX9+/fnw8++ICCggLAGBo3LCyMEydOsH37dsA4HwUFBYSEhLBjxw601hw7doxt27aVeqzLly9To0YNgoKCuHTpEsuWLQOMiSrq1avHypUrAeMXQtEcq5MmTeL111/Hy8uLNm3aWHEGbUMCXQgXFx0dTVhYGG3btuWee+65YghcW3niiSc4deoUYWFhvPzyy4SFhV3VtJGens6QIUOIioqiV69evPXWW4DxWORvv/1GZGQkYWFhfPjhh4AxecfSpUvp0KHDVTdFn3nmGaZOnUp0dDSWI8Y+9NBDNGzYsHi+0KVLl+Ll5cXixYt55JFHiIqKYsCAAeTm5tKrVy+Cg4MJDQ3lz3/+c/FMSiUFBgZy7733EhYWxuDBg4mNjS1e9+mnn/Lmm28SGRlJjx49uHDhAmBMwH3zzTfbbOIKa1k1fK5SahAwG3AD5mutZ5ZY3xxYANQDUoAJWuuk8vYpw+cKZyfD5/7BZDJhMpnw9vbm4MGDDBgwgIMHD+LuXj1bdbOysoiIiGDXrl3Url37uvdT0eFzr3m2lVJuwHtAfyAJ2KqUWqG1tvw31RvAIq31J0qpvsA/gT9d599BCOFkMjMz6devHyaTCa01c+fOrbZhvmbNGh588EGmTp16Q2F+Paw5452BQ1rrIwBKqSXACMAy0MOAp82v1wFf2bJIIUTV5u/vX2YbdHUzcOBATpw44ZBjW9OGHgyctHifZF5maRdwu/n1KKC2Uiqw5I6UUpOVUnFKqbiitiYhnJmjZvwSru96vlu2uin6F6CXUmoH0As4BRSU3EhrPU9rHaO1jqlXr56NDi2EY3h7e5OcnCyhLmxOa01ycnLxs+3WsqbJ5RTQ1OJ9E/Myy4OfxnyFrpTyBUZrrdMqVIkQTqZJkyYkJSUh/9oU9uDt7U2TJk0q9BlrAn0r0Fop1QIjyMcC4y03UEoFASla60LgWYwnXoRwaR4eHrRo0cLRZQhnc3Y3+DcF72v3WK2oaza5aK1NwOPAGmAvsFRrvUcpNUMpNdy8WW9gv1LqANAAeMXmlQohhLMqMMGer2DhbfBBd9jxX7scxqrn0O1BnkMXQri8zAuw/WOIWwgZp8C/GXR6EDpMgFoB17XLG3oOXQghRAWd2g5b5sHuZVCQBy37wJA3ofUAqGG/gbok0IUQwhZMeZD4FWyeC6fiwNMXou+FzpOh3s2VUoIEuhBC3IiMM7BtodGsknUeAlvB4Ncgahx416nUUiTQhRCiorSGk5uNq/G9K6CwAG4eCJ0fhJZ9oYwp/+xNAl0IIayVf9loF988F87Gg5cfxD4Mne6HgJaOrk4CXQghrintJGydD9sXweUUqB8GQ9+GyLvA0+fan68kEuhCCFEareHYeuNqfP8qY1nbIdD5IQjpAeYZjaoSCXQhhLCUlwW7lsCWD+HCXqgZAN2fgpj7jB6eVZgEuhBCACQfhq0fGb04c9OhURSMeB/CR4NHxQbJchQJdCFE9VVYCIfXwpa5cPAHo9NP2AijWaVp5yrZrFIeCXQhRPWTkw47Fxu9OVMOg28D6PUMxEyC2g2vf7f5BSRn5ZGSmcfFrFxSMvNIySr5Oo9Het3EoPDrP05ZJNCFENXHhf1GiO9aAnmZ0KQT9H7WuCp397xq85z8AlKy8kjOzCM5K9fidR7Jmeb3WeZ1mXlk5V01DQQAHm6KQB8vAnw8CfT1xNPdPlf+EuhCCNdWWAAHVhtBfuRntJsXaS2HcaTFeI57tyElPY+LPxwhxRzYF81X0smZueUGdICPJ4E+XgT6etI8sFbx60Afz+LgDvTxIsDXk9pe7qhKaL6RQBdCOK1cU8EVV80pWbnFr7PTLhB+7mtuSV9Bg8JznNWBLDKNYUlOX1IS6kBCDsbsmeBeQxHo60mAjxdB5oAO8PEkyNe4qjZeG+sDKzGgK0oCXQhRZZQX0CmZFs0b5m0yc01X7aOtOsEk9zWMcPsNb/LY6xnJN0GPcqphHwJ8azHVHNKBPp4Eml/X8a6aAV1REuhCCLvJMxWamzGMELZ8fUVomwP7UikBDcYV9B9Xyl40rVvryqvmmopWKT/TeP9/qHlmM9q9JipqPHR6kNCG4YRW8t/bUSTQhRBWKwroZPOVc/FNQcsbhBaBXV5A1y2+SvYksq6/8driqrmoPTrQx4s6Ncu4gi6aQGLDArh0Gvybw4B/oDpMgJp17XsyqiAJdCGqsTxTIanZJa+gSzR1mEM6OSuPSzmlB7RbDWXRjOFJhEVAB/heGdRB5QW0tU5tg83zYM+XxgQSN/U1xlZp3d+uE0hUdRLoQriQ/ILC4qtjyyvpko/cFTV9lBfQdWsVNWl4Eh7sV3yD8I8nOf54qqOOtwc1ati5DdqUa8zLuWXeHxNIdJxoTOlWSRNIVHUS6EJUYfkFhaRaPkp3jaaOjDICuobCCGBzILdrXOfK5g3LK2jfSgpoa2WcgbgFsO1j8wQSrWHw6xA1ttInkKjqJNCFqERFAZ1c3g1Ci8BOv5xf6n6MgPYs7qwS1rgOQear5gBfT/NrI6QDfTzxq1mFAtoaWsOJTcbV+BUTSEw25ud00AQSVZ0EuhA3wFRQSEp2nlU3CK0J6KI/oUVX0OaADrRonw7w8cLf2QLaWvmXIeELY2yVswngXTSBxAMQ0MLR1VV5EuhClJCalce5Sznm8TjySDHfEPzjWejc4ivstOyyA7puraIA9iS0UZ3i1yWbNwJ8vPCr6YGbKwa0tdJOWEwgkWqeQGIWRI6pUhNIVHUS6EKYpV/OZ+Z3e1m85eRV65SCgFoWV9AN65R5gzDAxxP/Wp7VO6CtoTUc/dVoVtm/ClDmCSQmV9kJJKo6CXQhgNW7z/LC17u5mJnLpO4hxDQPsLiCloC2qdxMiC+aQGIf1Ao0JpDodD/4NXF0dU5NAl1Ua+cycnjx6z2s3nOWsEZ1WDCxE+HBfo4uyzUlHzaaVXZ8ap5Aoj2MnAPtbneaCSSqOgl0US0VFmr+F3eSV1ftJc9UyLTBbbm/Rws83OTpCZsqLITDPxnzch76AWq4Q9hIiH3IGLpWmlVsSgJdVDtHLmTy7JcJbD6aQpeWAfzz9khaBMmNN5vKSYednxnNKkUTSPR+1ugIdAMTSIjySaCLaiO/oJB5vx5h9k8H8XKvwb9GRzAmpqlLjLJXZZzf98cEEvlZ0KQz9HkOQoeXOoGEsC0JdFEtxCel8cyyBPaeyWBweENeHt6O+nWk3dYmiiaQ2DwXjv4Cbl4QcQd0fhAad3B0ddWKBLpwadl5Jt7+4QAfbThKkK8XH0zoaJe5HKul7BTjufGtH0H6CajTBPq9ANH3gk+Qo6urlqwKdKXUIGA24AbM11rPLLG+GfAJ4G/eZprWepWNaxWiQjYcvMizy+M5mXKZ8bHNeGZQW/xqeji6LOd3NsG4Gk/4HEw5ENITBr4CbW4DN7lGdKRrnn2llBvwHtAfSAK2KqVWaK0TLTabDizVWs9RSoUBq4AQO9QrxDWlZefxj2/38sW2JFoE+bBkche6tAx0dFnOrSAf9q40bnKe2AgetYzBsTpPhgbtHF2dMLPm12ln4JDW+giAUmoJMAKwDHQNFA175gectmWRQlhDa8038Wd4eeUe0rLzeazPTTzRtzXeHtV3fOwblnnBGOUwzjyBRN0QGPAKdLi7Wk4gUdVZE+jBgGVf6CQgtsQ2LwHfK6WeAHyAW0vbkVJqMjAZoFmzZhWtVYgynUm/zN++2s2Pe88T2cSPRffFEtZYhla9bknbjAGy9iw3TyDRTyaQcAK2avAaB3ystX5TKdUV+I9SKlxrXWi5kdZ6HjAPICYmRtvo2KIaKyzUfLr5OP9avR9TYSHTh4QysVsI7tJBqOKKJ5CYa8wI5FkbOk4ynlYJau3o6oQVrAn0U0BTi/dNzMss3Q8MAtBa/66U8gaCgPO2KFKI0hw6f4lpyxKIO55Kz9ZBvDIygmaBtRxdlvPJOG0xgcSFPyaQaD8OvGo7ujpRAdYE+lagtVKqBUaQjwXGl9jmBNAP+FgpFQp4AxdsWagQRfJMhXzwy2H+vfYQNT3deOPOKEZHB0sHoYoonkBirnGzs7AAbh4EseYJJORcOqVrBrrW2qSUehxYg/FI4gKt9R6l1AwgTmu9Avgz8KFSagrGDdKJWmtpUhE2t/1EKtOWxXPgXCZDIxvx4rB21Kvt5eiynEf+ZeNxwy3zZAIJF2RVG7r5mfJVJZa9YPE6Eehu29KE+ENWrok3vt/PxxuP0bCON/PvieHWsAaOLst5pB6HuI8sJpBoB8NmQ8QY8JRmKlchvQBElffz/vM8v3w3p9Iuc0/X5kwd2Iba3tJB6Jq0Nrrib54HB74DFIQONZ4db95dmlVckAS6qLJSsvKYsXIPX+08zU31fPji4a7EhAQ4uqyqLzcTdi02OgFd3G9MINFjCsTcJxNIuDgJdFHlaK35eudpZnyTyKWcfJ7s15rH+tyEl7s8/1yu5MNGiO/8FHIzZAKJakgCXVQpSanZPL98N78cuED7pv78a3QkbRrKo3NlumoCCQ9oNxI6PwRNYqRZpZqRQBdVQkGhZtHvx3h9zX4AXhwWxj1dQ2Qez7LkpBtTuW39EFKOgG9D6P2ceQIJuVlcXUmgC4fbf/YSzyyLZ+fJNHrdXI9XRoXTpK48eVGqkhNINI2FPs/LBBICkEAXDpRrKuC9tYeY88thfL3cmXVXe0a0bywdhEoqLID93xmdgI7+ap5A4k7zBBLtHV2dqEIk0IVDxB1L4Zll8Ry+kMWoDsFMHxJKoK90ELpCqRNIvGieQEKGAxZXk0AXlepSTj6vrd7PfzYdJ9i/Jh9P6kTvNvUdXVbVcibeuBpP+OKPCSQGvQo3D5YJJES55NshKs1Pe88x/avdnM3IYVL3EP4yoA0+XvIVBMwTSKwwTyDxu3kCiXHmCSTCHF2dcBLyf5OwuwuXcnl55R6+iT9Dmwa1ef/uaDo0k8kRAMg8bzGBxBmo2wIGvgrtx8sEEqLCJNCF3Wit+WJbEv/4di+X8wp4uv/NPNzrJjzdZaxykuKMp1UsJ5AYNhta9Ycacn7E9ZFAF3ZxIjmb55YnsOHQRWKa12Xm6Aha1a/mHYRMuUaAb54Lp7cbE0jE3GeMdCgTSAgbkEAXNmUqKGThb8d484f9uNeowd9HhnN352bUqM4dhNJP/TGBRPZFCLoZbnvDmGRZJpAQNiSBLmwm8XQG076MJz4pnX5t6/P3keE09q/p6LIcQ2vj5uZm8wQSuhDaDDZucrbsLV3yhV1IoIsblpNfwDs/HWTur0eoW8uDf4/vwJCIRtWzg1BetnkCiQ/hXAJ4+0PXR41mlbohjq5OuDgJdHFDNh1J5tkvEzh6MYs7OjZh+pBQ/GtVwy7oqcdh63yjI1BOGjQIh2HvGD06ZQIJUUkk0MV1Sb+cz8zv9rF4ywmaBtTkv/fH0qN1kKPLqlxaw5GfjadV9n8HqoZ5AomHoHk3aVYRlU4CXVTY6t1neeHr3VzMzGXyLS156tbW1PKsRl+lqyaQCIKefzZPIBHs6OpENVaN/i8UN+p8Rg4vfL2H1XvOEtqoDvPvjSGyib+jy6o8yYeNq/GdnxkTSDTuACM/gHajZAIJUSVIoItr0lrzv60neWXVXnJNhfx1UBse7NkSD7dq0AGmsBAO/WiMrXLoR/MEEqMg1jyBhBBViAS6KNfRi1k8+2U8m46kENsigH/eHkHLer6OLsv+LqcZU7lt+RBSj8oEEsIpSKCLUuUXFDJ//VFm/XgAT/ca/PP2CO6Kaer6HYSSD8Pv/4Zd/zNPINEF+v3NmEDCzcPR1QlRLgl0cZWEpHSeWRZP4pkMBrZrwIwR4TSo4+JtxGcTYP1bkPiV0awScSfEToZGUY6uTAirSaCLYpfzCnj7xwPMX3+EQF8vPpgQzaDwRo4uy75ObDKC/OAaY2yV7v8HXR4FXxmjXTgfCXQBwG+HLvLslwmcSMlmXOemTBscil9NF21i0BoO/2QE+fHfoFYg9J0OnR6EmtXoqR3hciTQq7m07Dxe+XYvn29LIiSwFp89GEu3m1y0g1BhgTGuyvo34Ww81AmGQf+C6HukN6dwCRLo1ZTWmlUJZ3lxxR5Ss/N4pPdN/F+/1nh7uDm6NNsz5UHCUtgwC5IPQmArGP5viLwL3KvhMAXCZUmgV0Nn0i/zt6/28OPec4QH1+GT+zrRrrGfo8uyvbxs2PEf+O0dyEiChhFw58fGEys1XPAXl6j2JNCrkcJCzadbTvCv7/ZhKizkudvacl/3Fri7Wgehy2nGQFmb5hjjjzfrZp4NqJ+MryJcmgR6NXHofCbPfhnP1mOpdG8VyKujImge6OPosmwr8wJset8I89wMYzq3nk8bA2UJUQ1YFehKqUHAbMANmK+1nlli/dtAH/PbWkB9rbU8LlAF5JkKmfvLYd5de4ianm68dkckd3Zs4lpjlaedgI3vGkPXmnKh3UjoMUWeIRfVzjUDXSnlBrwH9AeSgK1KqRVa68SibbTWUyy2fwLoYIdaRemWx90AABVMSURBVAXtPJnGtGXx7Dt7iSGRjXhxWBj1a7tQB6ELB2DD28YNT5QxpVv3pyColaMrE8IhrLlC7wwc0lofAVBKLQFGAIllbD8OeNE25YnrkZVr4s3vD7Bw41Ea1Pbmw3ti6B/mQuOPnN5hPEO+dyW4exvPj3d7HPyaOLoyIRzKmkAPBk5avE8CYkvbUCnVHGgBrL3x0sT1+OXABZ77MoFTaZeZ0KUZfx3UljreLtBBSGujE9D6N+HwWvDyg1v+ArEPg4+LPjcvRAXZ+qboWOALrXVBaSuVUpOByQDNmjWz8aGrt5SsPP7xTSJf7jhFy3o+fP5wVzqFBDi6rBunNRxYAxvegpObwac+3PoSxNwP3nUcXZ0QVYo1gX4KaGrxvol5WWnGAo+VtSOt9TxgHkBMTIy2skZRDq01K3ad5uWViWRczueJvq14rE8r5+8gVFgAe5YbbeTndoNfM7jtDegwATxqOro6IaokawJ9K9BaKdUCI8jHAuNLbqSUagvUBX63aYWiTKfSLjN9eQLr9l8gqqk//xodQduGTn7Vaso1pnfbMMsYhzyoDYyaC+GjZfhaIa7hmoGutTYppR4H1mA8trhAa71HKTUDiNNarzBvOhZYorWWK287KyjU/Of3Y7y2Zj9awwtDw7i3WwhuzjxWeW4mbP/EePzw0hljercB/4U2Q6CGi3V8EsJOrGpD11qvAlaVWPZCifcv2a4sUZYD5y7xzLJ4dpxI45ab6/HKyHCaBjjxwFLZKcasQJvnwOVUCOkJI+dAy97Sq1OICpKeok4i11TA++sO8/7Ph/D1cuftu6IY2T7YeTsIXTprzAwUtxDyMqHNbdDjaWjaydGVCeG0JNCdwLbjKTyzLIFD5zMZ0b4xLwwNI9DXy9FlXZ+Uo7DxHdjxKRTmG23jPaZAg3aOrkwIpyeBXoVl5pp4bfU+/rPpOI3qeLNwYif6tHXSmXTOJRpPrOxeZox02P5u6P4kBLR0dGVCuAwJ9Cpq7b5zPL98N2czcri3awh/GdgGXy8n/HElxRm9Ovd/Cx4+0PVR6PIY1HHxqe2EcAAnTAjXdjEzl5dXJrJy12la1/fli4e70bF5XUeXVTFaw9FfjF6dR38Fb3/o/Sx0ngy1XKCzkxBVlAR6FaG15svtp/j7t4lk5ZqYcuvNPNL7JjzdneiRvcJC2L/KCPLT28G3IQx4BTpOBC9fR1cnhMuTQK8CTqZk89zyBNYfvEjH5nWZeXsErRvUdnRZ1iswGW3jG96CC/ugbggMnQXtx4O7k968FcIJSaA7UEGhZuFvR3nz+wPUUDBjRDsmxDanhrN0EMrPgZ3/hd9mG2OS1w+D0R9B2Ehwk6+WEJVN/q9zkL1nMpi2LJ5dSen0bVuff4wMp7G/k4xRkpMBcQvg9/cg6zw06QSDX4PWA6VXpxAOJIFeyXLyC3h37UHm/nIEv5oevDOuA8MiGzlHB6GsZNj8AWyZCznp0LIP9FwAIT2kV6cQVYAEeiXafCSZZ79M4MjFLEZHN2H6kFDq+ng6uqxrSz9l9Orc9jHkZ0PoMKNXZ3C0oysTQliQQK8EGTn5zPxuH59tPkGTujX5z/2d6dm6nqPLurbkw0ZnoF1LQBdC5Bhjirf6bR1dmRCiFBLodrZmz1le+Ho3Fy7l8kCPFjw94GZqeVbx0342wegMlPgV1PAwHjvs9gTUbe7oyoQQ5ajiyeK8zl/K4aUVe1iVcJa2DWsz708xRDX1d3RZ5TuxyXiG/OD34Fkbuj0JXR6F2i40H6kQLkwC3ca01iyNO8kr3+4lx1TI1IFtmHxLSzzcqujTH1rD4Z+MK/Ljv0GtQOg73Zh4uWYV/wUkhLiCBLoNHbuYxbNfJvD7kWQ6twjgn7dHcFO9KtpDsrAA9q40rsjPxkOdYBj0L4j+E3j6OLo6IcR1kEC3AVNBIfM3HOXtHw7g6VaDV0dFMLZT06rZQciUBwlLjZudyYcgsBUM/zdE3gXuTvDEjRCiTBLoN2j3qXSeWRbPntMZ9A9rwN9HhNPQz9vRZV0tLxt2/Ad+ewcykqBhBNz5MYQON4azFUI4PQn063Q5r4BZPx1g/vqjBPh4MufuaAaFN6x6HYQup8HW+bBpDmRfhGbdYNhsaNVPOgMJ4WIk0K/DxkMXeXZ5AseTs7krpinP3RaKX60qNiN95nnY9D5s/QhyM6BVf+j5NDTv5ujKhBB2IoFeAenZ+byyKpGlcUk0D6zFZw/E0q1VkKPLulLaCdj4LmxfBKZcaDfSmOKtUZSjKxNC2JkEuhW01ny3+ywvfL2H1Ow8Hu51E0/d2hpvjyrU9nxhP2yYZdzwREHUWKNXZ1ArR1cmhKgkEujXcDY9h799vZsfEs/RrnEdPp7UifBgP0eX9YfTO4xnyPeuBHdv4/nxbo+DXxNHVyaEqGQS6GUoLNQs3nqCmav2kVdQyLOD23J/jxa4V4UOQlobnYDWvwmH14KXH9zyF4h9GHyqWBOQEKLSSKCX4vCFTJ5dlsCWYyl0uymQV0dFEBJUBTrbaA0H1hhBnrQFfOrDrS9BzP3gXcfR1QkhHEwC3UJ+QSFzfznMO2sP4e1eg9dGR3JnTBPHP4pYYDIGytrwNpzbDX7N4LY3oMME8HCSSTGEEHYngW6262QazyyLZ9/ZSwyJaMSLw8OoX9vBHYRMubBrsXGzM/UoBLWBUXMhfDS4VbHHJIUQDlftAz07z8Sb3x9g4W9HqVfbi3l/6siAdg0dW1RupjGZxO//hktnoHEHGPBfaDNEpngTQpSpWgf6rwcu8NzyBJJSL3N3bDOeGdyWOt4OvPLNToEtH8LmOXA5FUJ6wsg50LK39OoUQlxTtQz01Kw8/v5tIl9uP0XLIB+WPtSVzi0CHFfQpbPG1XjcQsjLhDa3GVO8Ne3kuJqEEE6nWgW61poVu04zY2Ui6ZfzebxPKx7v28pxHYRSjsLGd2DHf6HQZLSN95gCDdo5ph4hhFOrNoF+Ku0yf/tqN2v3nSeqiR//fSCW0EYOetTvXKLxxMruZcZIh+3vhu5PQkBLx9QjhHAJVgW6UmoQMBtwA+ZrrWeWss0Y4CVAA7u01uNtWOd1KyzU/GfTcV5bvY9CDdOHhDKpewvcHDFWeVKc8Qz5/lXg4QNdH4Uuj0GdRpVfixDC5Vwz0JVSbsB7QH8gCdiqlFqhtU602KY18CzQXWudqpSqb6+CK+LguUs8syye7SfS6Nk6iFdHRdA0oFblFqE1HPkZNrwFR38Fb3/o/Sx0ngy1HNhuL4RwOdZcoXcGDmmtjwAopZYAI4BEi20eBN7TWqcCaK3P27rQisg1FTDn58O8t+4QPl7uvDUmilEdgiu3g1BhoXElvv5NOL0dfBvCgH9Ax4ngVbvy6hBCVBvWBHowcNLifRIQW2KbmwGUUr9hNMu8pLVeXXJHSqnJwGSAZs2aXU+917TteCrTlsVz8Hwmw6Ma88KwMIJ8vexyrFIV5Btt4xvehgv7oG4IDJ0F7ceDeyXWIYSodmx1U9QdaA30BpoAvyqlIrTWaZYbaa3nAfMAYmJitI2ODUBmrok31uznk9+P0aiONwsmxtC3bQNbHqJ8+Tmw87/w22xjTPL6YTD6IwgbCW7V5t6zEMKBrEmaU0BTi/dNzMssJQGbtdb5wFGl1AGMgN9qkyqvYd2+8zy/PIEzGTnc06U5Uwe1xderkkI0JwPiFsDv70HWeWjSCQa/Bq0HSq9OIUSlsib1tgKtlVItMIJ8LFDyCZavgHHAQqVUEEYTzBFbFlqa5MxcXl6ZyIpdp2lV35cvHu5Kx+aVdKMxKxk2fwBb5kJOOrTsAz0XQEgP6dUphHCIawa61tqklHocWIPRPr5Aa71HKTUDiNNarzCvG6CUSgQKgKla62R7Fa21ZvmOU/z9m0Qyc008dWtrHul9E17uldBBKP2U0atz28eQnw2hw4xencHR9j+2EEKUQ2lt06Zsq8XExOi4uLgKf+5kSjbPLU9g/cGLdGjmz79GR3Jzg0p4aiT5sHGjc9cS0IUQOcaY4q1+W/sfWwghzJRS27TWMaWtc7q7dSvjT7P9eCovD2/HhC7N7d9B6Ey8EeSJX0END+Oxw25PQN3m9j2uEEJUkNMF+oM9WzKyfTCN/e08scOJTcYz5Ae/B8/a0O1J6PIo1K7EJ2eEEKICnC7QPdxq2C/MtYZDPxlBfmIj1AqEvtONiZdr+tvnmEIIYSNOF+h2UVgAe1caQX42HuoEw6B/QfSfwLMKzCUqhBBWqN6BbsqDhKVGG3nyIQhsBcP/DZF3gbuno6sTQogKqZ6BnpcN2xfBxnchIwkaRsCdH0PocGM4WyGEcELVK9Avp8HW+bBpDmRfhGbdYNhsaNVPOgMJIZxe9Qj0zPOw6X3Y+hHkZkCr/tDzaWjezdGVCSGEzbh2oKedMJpVti8CUy60G2lM8dYoytGVCSGEzblmoF/YDxtmGTc8URA11ujVGdTK0ZUJIYTduFagn9puzAy09xtw9zaeH+/2OPg1cXRlQghhd84f6FrD8d+MZ8gPrwUvP7jlLxD7MPgEObo6IYSoNM4b6FrDgTVGkCdtAZ/6cOtLEHM/eNdxdHVCCFHpnC/QC0zGQFkb3oZzu8GvGdz2BnSYAB52Ht9FCCGqMOcL9F9mwq+vQ1AbGDUXwkeDm4ejqxJCCIdzvkCPvsd47LDNEJniTQghLDhfoPs3M/4IIYS4glziCiGEi5BAF0IIFyGBLoQQLkICXQghXIQEuhBCuAgJdCGEcBES6EII4SIk0IUQwkVIoAshhIuQQBdCCBchgS6EEC5CAl0IIVyEBLoQQrgICXQhhHARVgW6UmqQUmq/UuqQUmpaKesnKqUuKKV2mv88YPtShRBClOea46ErpdyA94D+QBKwVSm1QmudWGLT/2mtH7dDjUIIIaxgzRV6Z+CQ1vqI1joPWAKMsG9ZQgghKsqaQA8GTlq8TzIvK2m0UipeKfWFUqppaTtSSk1WSsUppeIuXLhwHeUKIYQoi61uiq4EQrTWkcAPwCelbaS1nqe1jtFax9SrV89GhxZCCAHWBfopwPKKu4l5WTGtdbLWOtf8dj7Q0TblCSGEsJY1gb4VaK2UaqGU8gTGAissN1BKNbJ4OxzYa7sShRBCWOOaT7lorU1KqceBNYAbsEBrvUcpNQOI01qvAJ5USg0HTEAKMNGONQshhCiF0lo75MAxMTE6Li7OIccWQghnpZTaprWOKW2d9BQVQggXIYEuhBAuQgJdCCFchAS6EEK4CAl0IYRwERLoQgjhIiTQhRDCRUigCyGEi5BAF0IIFyGBLoQQLkICXQghXIQEuhBCuAgJdCGEcBES6EII4SIk0IUQwkU4bDx0pdQF4Ph1fjwIuGjDcmxF6qoYqaviqmptUlfF3EhdzbXWpU7K7LBAvxFKqbiyBnh3JKmrYqSuiquqtUldFWOvuqTJRQghXIQEuhBCuAhnDfR5ji6gDFJXxUhdFVdVa5O6KsYudTllG7oQQoirOesVuhBCiBIk0IUQwkVUuUBXSg1SSu1XSh1SSk0rZb2XUup/5vWblVIhFuueNS/fr5QaWMl1Pa2USlRKxSulflJKNbdYV6CU2mn+s6KS65qolLpgcfwHLNbdq5Q6aP5zbyXX9bZFTQeUUmkW6+x5vhYopc4rpXaXsV4ppd4x1x2vlIq2WGeX82VFTXeba0lQSm1USkVZrDtmXr5TKRVnq5oqUFtvpVS6xc/rBYt15X4H7FzXVIuadpu/UwHmdXY5Z0qppkqpdeYc2KOU+r9StrHv90trXWX+AG7AYaAl4AnsAsJKbPMo8IH59Vjgf+bXYebtvYAW5v24VWJdfYBa5tePFNVlfp/pwPM1Efh3KZ8NAI6Y/1vX/LpuZdVVYvsngAX2Pl/mfd8CRAO7y1h/G/AdoIAuwOZKOF/Xqqlb0bGAwUU1md8fA4IceL56A9/c6HfA1nWV2HYYsNbe5wxoBESbX9cGDpTy/6Ndv19V7Qq9M3BIa31Ea50HLAFGlNhmBPCJ+fUXQD+llDIvX6K1ztVaHwUOmfdXKXVprddprbPNbzcBTWx07BuqqxwDgR+01ila61TgB2CQg+oaByy20bHLpbX+FUgpZ5MRwCJt2AT4K6UaYcfzda2atNYbzceEyvtuFR37WuerLDfy3bR1XZXy/dJan9Fabze/vgTsBYJLbGbX71dVC/Rg4KTF+ySuPiHF22itTUA6EGjlZ+1Zl6X7MX4LF/FWSsUppTYppUbaqKaK1DXa/M+7L5RSTSv4WXvWhblpqgWw1mKxvc6XNcqq3Z7nqyJKfrc08L1SaptSarID6gHoqpTapZT6TinVzrysSpwvpVQtjGBcZrHY7udMGU3BHYDNJVbZ9fvlXtEPiPIppSYAMUAvi8XNtdanlFItgbVKqQSt9eFKKmklsFhrnauUegjjXzd9K+nY1hgLfKG1LrBY5sjzVWUppfpgBHoPi8U9zOeqPvCDUmqf+eq1smzH+HllKqVuA74CWlfi8a9lGPCb1tryat6u50wp5YvxC+QprXWGrfZrjap2hX4KaGrxvol5WanbKKXcAT8g2crP2rMulFK3As8Dw7XWuUXLtdanzP89AvyM8Zu7UurSWidb1DIf6GjtZ+1Zl4WxlPjnsB3PlzXKqt2e5+ualFKRGD+/EVrr5KLlFufqPLAc2zUzWkVrnaG1zjS/XgV4KKWCcPD5slDe98vm50wp5YER5p9qrb8sZRP7fr9sfWPgBm8quGPcDGjBHzdS2pXY5jGuvCm61Py6HVfeFD2C7W6KWlNXB4ybQK1LLK8LeJlfBwEHsdHNISvramTxehSwSf9xE+aoub665tcBlVWXebu2GDeoVGWcL4tjhFD2Tb4hXHnTaou9z5cVNTXDuCfUrcRyH6C2xeuNwCBbnisramtY9PPDCMYT5nNn1XfAXnWZ1/thtLP7VMY5M/+9FwGzytnGrt8vm/7gbXRSbsO4O3wYeN68bAbGVS+AN/C5+Qu+BWhp8dnnzZ/bDwyu5Lp+BM4BO81/VpiXdwMSzF/oBOD+Sq7rn8Ae8/HXAW0tPnuf+TweAiZVZl3m9y8BM0t8zt7nazFwBsjHaKe8H3gYeNi8XgHvmetOAGLsfb6sqGk+kGrx3YozL29pPk+7zD/j5215rqys7XGL79cmLH7plPYdqKy6zNtMxHhQwvJzdjtnGE1hGoi3+FndVpnfL+n6L4QQLqKqtaELIYS4ThLoQgjhIiTQhRDCRUigCyGEi5BAF0IIFyGBLoQQLkICXQghXMT/AyRTsv2tNaIGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_history(lstm_history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we're getting somewhere! (But still not very far.)\n",
    "\n",
    "Recall that when generating the data, we applied `padding='post'`. Let's see if that made a difference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pre-padded:\n",
      "[    0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     9   215 20597  1719    20     5\n",
      "   192   592    14     1    82    54     2     9    24     5   131     9\n",
      "    62   254     8     5    26   175    48    43    21    22     3   333\n",
      "     4  7261 13419    21    76   115     8     1  2931    16   300   269\n",
      "  1000    23   326    38   123     8     1   112    35   312   330     6\n",
      "     3   113  3308     2  5073     2    45     4     1   135    96    26\n",
      "   601   176    17     8   491    42     7     1   126     1  2252    19\n",
      "     1   284    22    39    13    83    13     1    18    43    20  2780\n",
      "    14   144  8057   447     8    44  3395     4 14895   384    35     1\n",
      "   266 13420     2     3    83  2695    15     1   163   255    33    44\n",
      "   820   139   227    31  1720    18    76   115     5   102  8058     1\n",
      "   163  8057     2   312   330    19     1    51   388   487   961   266\n",
      "   349     5  4009    42    10   113 12257   943    18     8    12   180\n",
      "   733     2   142  1259     5    24     8     7    57  1587]\n"
     ]
    }
   ],
   "source": [
    "pre_padded_X = pad_sequences(\n",
    "    sequences,\n",
    "    maxlen=250,\n",
    "    padding='pre',\n",
    "    truncating='pre',\n",
    "    value=0\n",
    ")\n",
    "\n",
    "print('Pre-padded:')\n",
    "print(pre_padded_X[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The real (non-pad) tokens are now aligned with the end of the sequence, leaving less distance from the useful part of the input to the point where the model sees the label.\n",
    "\n",
    "(Note that this is effectively what [masking](https://www.tensorflow.org/guide/keras/masking_and_padding) would do here.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 22500 samples, validate on 2500 samples\n",
      "Epoch 1/3\n",
      "22500/22500 [==============================] - 1101s 49ms/sample - loss: 0.4533 - acc: 0.7888 - val_loss: 0.3423 - val_acc: 0.8612\n",
      "Epoch 2/3\n",
      "22500/22500 [==============================] - 1330s 59ms/sample - loss: 0.2108 - acc: 0.9196 - val_loss: 0.3435 - val_acc: 0.8692\n",
      "Epoch 3/3\n",
      "22500/22500 [==============================] - 1011s 45ms/sample - loss: 0.0950 - acc: 0.9654 - val_loss: 0.4258 - val_acc: 0.8248\n"
     ]
    }
   ],
   "source": [
    "model = build_rnn_model(LSTM, sequence_length, vocab_size, num_classes)\n",
    "\n",
    "model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "\n",
    "lstm_history_pre = model.fit(pre_padded_X, Y, epochs=epochs, batch_size=batch_size, validation_split=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "History: [0.8612, 0.8692, 0.8248]\n",
      "Max accuracy: 0.8692\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXhU5dn48e+dycqWhUSUhE1lC1kgRBZREZHNohRXQK1QFaWir/iWSou/qtiFt1qXWoorVt9XoSrVorKoBQuWogSULQgBREmCLIGENcskz++Pc2YyCVkmMJNJJvfnunJx5jznzNw5Ge45c5/nPI8YY1BKKRW8QgIdgFJKKf/SRK+UUkFOE71SSgU5TfRKKRXkNNErpVSQ00SvlFJBrt5ELyILROSgiGytpV1E5E8isktENotIhkfbHSKSY//c4cvAlVJKecebM/q/AqPraB8DdLd/pgLzAUQkDngUGAgMAB4VkdhzCVYppVTD1ZvojTGrgSN1bDIOeMNY1gExInIBMAr4xBhzxBhzFPiEuj8wlFJK+UGoD54jEdjn8TjXXlfb+jOIyFSsbwO0bt26f69evXwQllJKtRwbNmw4bIxJqKnNF4n+nBljXgJeAsjMzDRZWVkBjkgppZoXEfmutjZf9LrJAzp5PE6y19W2XimlVCPyRaJfAvzE7n0zCCgyxuwHVgAjRSTWvgg70l6nlFKqEdVbuhGRhcCVQLyI5GL1pAkDMMa8ACwFrgF2AaeAKXbbERF5AlhvP9UcY0xdF3WVUkr5Qb2J3hgzsZ52A9xXS9sCYMHZhVaprKyM3NxciouLz/WplDpDZGQkSUlJhIWFBToUpfyiSVyMrU9ubi5t27ala9euiEigw1FBxBhDQUEBubm5dOvWLdDhKOUXzWIIhOLiYtq3b69JXvmciNC+fXv9tqiCWrNI9IAmeeU3+t5Swa7ZJHqllFJnRxO9F4YNG8aKFVV7hj777LNMmzatzv3atGkDQH5+PjfeeGON21x55ZXUd4PYs88+y6lTp9yPr7nmGgoLC70J3Wf27t3LW2+91aivqZTyDU30Xpg4cSKLFi2qsm7RokVMnFhnhyS3jh078u67757161dP9EuXLiUmJuasn+9sNJVE73Q6Ax2CUs2OJnov3HjjjXz00UeUlpYCVtLLz8/n8ssv58SJEwwfPpyMjAxSU1P5xz/+ccb+e/fuJSUlBYDTp08zYcIEevfuzfjx4zl9+rR7u2nTppGZmUmfPn149NFHAfjTn/5Efn4+w4YNY9iwYQB07dqVw4cPA/D000+TkpJCSkoKzz77rPv1evfuzd13302fPn0YOXJklddxeeedd0hJSSE9PZ0rrrgCgPLycmbOnMkll1xCWloaL774IgCzZs1izZo19O3bl2eeeabK89R1DN544w3S0tJIT0/n9ttvB+DAgQOMHz+e9PR00tPTWbt2bZVjBPDUU0/x2GOPAda3ngcffJDMzEyee+45PvjgAwYOHEi/fv24+uqrOXDggDuOKVOmkJqaSlpaGosXL2bBggU8+OCD7ud9+eWXmTFjRh1/baWCT7PoXunp8Q+2kZ1/zKfPmdyxHY9e26fW9ri4OAYMGMCyZcsYN24cixYt4uabb0ZEiIyM5L333qNdu3YcPnyYQYMGcd1119V6gW/+/Pm0atWK7du3s3nzZjIy3MP389vf/pa4uDjKy8sZPnw4mzdv5oEHHuDpp59m1apVxMfHV3muDRs28Nprr/HFF19gjGHgwIEMHTqU2NhYcnJyWLhwIS+//DI333wzixcv5rbbbquy/5w5c1ixYgWJiYnuUtCrr75KdHQ069evp6SkhCFDhjBy5Ejmzp3LU089xYcffnjG71TbMcjOzuY3v/kNa9euJT4+niNHrPvlHnjgAYYOHcp7771HeXk5J06c4OjRo3X+jUpLS90lrqNHj7Ju3TpEhFdeeYU//OEP/PGPf+SJJ54gOjqaLVu2uLcLCwvjt7/9LU8++SRhYWG89tpr7g8vpVqKZpfoA8VVvnEl+ldffRWw+mH/6le/YvXq1YSEhJCXl8eBAwc4//zza3ye1atX88ADDwCQlpZGWlqau+3tt9/mpZdewul0sn//frKzs6u0V/f5558zfvx4WrduDcD111/PmjVruO666+jWrRt9+/YFoH///uzdu/eM/YcMGcLkyZO5+eabuf766wH4+OOP2bx5s7vUVFRURE5ODuHh4bXGUdsxWLlyJTfddJP7AyouLg6AlStX8sYbbwDgcDiIjo6uN9Hfcsst7uXc3FxuueUW9u/fT2lpqbv/+6efflqlxBYba01/cNVVV/Hhhx/Su3dvysrKSE1NrfO1lAo2zS7R13Xm7U/jxo1jxowZbNy4kVOnTtG/f38A3nzzTQ4dOsSGDRsICwuja9euZ9Un+9tvv+Wpp55i/fr1xMbGMnny5HPq2x0REeFedjgcNZZuXnjhBb744gs++ugj+vfvz4YNGzDG8PzzzzNq1Kgq23722We1vpYvjkFoaCgVFRXux9X3d32YAdx///089NBDXHfddXz22WfuEk9t7rrrLn73u9/Rq1cvpkyZ0qC4lAoGWqP3Ups2bRg2bBg//elPq1yELSoq4rzzziMsLIxVq1bx3Xe1jhQKwBVXXOG+qLl161Y2b94MwLFjx2jdujXR0dEcOHCAZcuWufdp27Ytx48fP+O5Lr/8ct5//31OnTrFyZMnee+997j88su9/p12797NwIEDmTNnDgkJCezbt49Ro0Yxf/58ysrKANi5cycnT56sNYa6jsFVV13FO++8Q0FBAYC7dDN8+HDmz58PWNcEioqK6NChAwcPHqSgoICSkpIaS0Ser5eYaE1t8Prrr7vXjxgxgnnz5rkfu74lDBw4kH379vHWW295fQFdqWCiib4BJk6cyKZNm6oki1tvvZWsrCxSU1N54403qG/SlGnTpnHixAl69+7Nr3/9a/c3g/T0dPr160evXr2YNGkSQ4YMce8zdepURo8e7b4Y65KRkcHkyZMZMGAAAwcO5K677qJfv35e/z4zZ84kNTWVlJQULr30UtLT07nrrrtITk4mIyODlJQU7rnnHpxOJ2lpaTgcDtLT08+4GFvbMejTpw+zZ89m6NChpKen89BDDwHw3HPPsWrVKlJTU+nfvz/Z2dmEhYXx61//mgEDBjBixIg6j+Njjz3GTTfdRP/+/atct3jkkUc4evSo+wLzqlWr3G0333wzQ4YMcZdzlGpJxBqTrOmoaeKR7du307t37wBFpILB2LFjmTFjBsOHD6+xXd9jqrkTkQ3GmMya2vSMXgW1wsJCevToQVRUVK1JXqlg1+wuxirVEDExMezcuTPQYSgVUHpGr5RSQU4TvVJKBTlN9Eop1RQcy4cftvrlqbVGr5RSje10IeR/BfkbIW8j5G2A4/uhYwZMXVX//g2kid4LBQUF7h4bP/zwAw6Hg4SEBAC+/PLLOocHcJkyZQqzZs2iZ8+etW4zb948YmJiuPXWW30TuJdWrlxJq1atGDRoUKO+rlItQlkxHNhamdDzNkBBTmV7+4uh2xWQ2B+Sauwdec400Xuhffv2fP3114B1s06bNm34+c9/XmUbYwzGGEJCaq6Gvfbaa/W+zn331TjHut+tXLmS+Pj4gCf68vJyHA5HQGNQ6pxUVMDhnfaZup3Uf9gKFdad5rTpAImZkD7BSuwd+0GU/4cc1xr9Odi1axfJycnceuut9OnTh/379zN16lT3UMNz5sxxb3vZZZfx9ddf43Q6iYmJYdasWaSnpzN48GAOHjwIWHd2uoYavuyyy5g1axYDBgygZ8+erF27FoCTJ09yww03kJyczI033khmZqb7Q8jTzJkzSU5OJi0tjYcffhiwhge+/vrryczMZMCAAaxbt47du3fzyiuv8OSTT9K3b1/367isW7eOwYMH069fP4YMGUJOjnUm4nQ6mTFjBikpKaSlpfGXv/wFgC+++ILBgweTnp7OwIEDOXXqFK+88kqVoYJHjx7N559/7j4WDz74IGlpaXz55Zc8+uijXHLJJaSkpHDvvffiuqFv586dXHXVVaSnp5ORkcHevXuZNGlSlaESbrnlFj766KNz+6Mq5S1joCgXspfAJ4/CX8fC3M7wl4Hw/jTY9DeIaAuXTodb/g9mZMN/74CJb8EVP4eLhjVKkofmeEa/bBb8sMW3z3l+KoyZe1a7fvPNN7zxxhtkZlpfuebOnUtcXBxOp5Nhw4Zx4403kpycXGWfoqIihg4dyty5c3nooYdYsGABs2bNOuO5jTF8+eWXLFmyhDlz5rB8+XKef/55zj//fBYvXsymTZuqDHPscuDAAZYuXcq2bdsQEfcQxA888AC/+MUvGDRoEHv37mXs2LFs3bqVu+66i/j4+CrJ2KV3796sWbOG0NBQli9fziOPPMLf/vY35s+fT35+Pps2bcLhcHDkyBGKi4uZMGECixcvJiMjg6KioiqDq9WkqKiIK664wv0B17NnTx5//HGMMUyaNInly5czZswYJk6cyGOPPca1115LcXExFRUV3HnnncyfP5+xY8dy9OhR1q9f3yQmR1FB6vRRq66et6GyDHPCmguBkDArj7jO1BMzoH13qOUbfmNrfom+ibnooovcSR5g4cKFvPrqqzidTvLz88nOzj4j0UdFRTFmzBjAGkJ4zZo1NT63a+hgz2GGP//8c/cZenp6On36nDmaZ1xcHCEhIdx999386Ec/YuzYsYA1jO+OHTvc2x09erTGUS09FRYW8pOf/ITdu3dXWf/pp5/y4IMPukstcXFxfPXVV3Tu3Nn94RMdHV3ncwOEh4czfvx49+N//vOfPPnkkxQXF3P48GH69+/PoEGDOHz4MNdeey1gjX8P1qBp06dPp6CggIULF3LzzTdr6Uf5RlmxdULpKr/kbYAjHv8H4nvARVfZ5ZcMOD8FQus+qQmk5pfoz/LM2188h8/Nycnhueee48svvyQmJobbbrutxuF6PS/eOhyOWqfHc50N17VNTcLCwsjKyuKTTz7hnXfeYf78+Xz88cfubwjeXDx2mT17NqNGjeJnP/sZu3btYvTo0V7v61LXEMRRUVHuSVpOnTrF9OnT2bhxI4mJiTzyyCN1DncsItx222289dZbvP7667z55psNjk0pKsqturpnUj+wDSrs/3NtO1pn6P1us/7t2A8i6z+JaUqaxveKIHHs2DHatm1Lu3bt2L9//xkTivvCkCFDePvttwHYsmUL2dnZZ2xz/Phxjh07xtixY3nmmWf46quvALj66qurDOPrqu3XNwSxa0jgv/71r+71I0aM4IUXXqC8vBywhiBOTk7m+++/Z+PGjYB1PMrLy+natStfffUVxhj27t3Lhg0banyt06dPExISQnx8PMePH2fx4sWANYFIQkICH3zwAWB9ULjm0J0yZQpPPvkkERERdfZoUgqw6uqF+2Db+/Dx/4PXfmTX1QfBP+6DLYshKhaG/Bfc8iY8tB3+eztMeBMufwguvLLZJXnw8oxeREYDzwEO4BVjzNxq7V2ABUACcAS4zRiTa7eVA66i+vfGmOt8FHuTk5GRQXJyMr169aJLly5Vhhr2lfvvv5+f/OQnJCcnu3+ql0iKioq4/vrrKSkpoaKigqeffhqwum9OmzaN1157zX0NYd68eYwbN46bbrqJv//978ybN49LL73U/VwPP/wwP/3pT3n88cfd5SaAe+65h5ycHNLS0ggNDWXatGnce++9LFy4kGnTplFcXExUVBQrV65k6NChJCYm0rt3b/r06eOe+aq69u3bc8cdd5CcnMwFF1zAwIED3W1vvvkm99xzD7NnzyY8PJzFixfTpUsXOnbsSI8ePZgwYYIvD7MKFqeOVO2rnrcBTh6y2hzhcH4a9J1k19X7Q9xFTaau7kv1DlMsIg5gJzACyAXWAxONMdke27wDfGiMeV1ErgKmGGNut9tOGGPaeBuQDlNcN6fTidPpJDIykpycHEaOHElOTg6hoc2vCucLJ0+eJDU1lU2bNtG2bduzfh59jwWBstOwf3PVEszRb+1Gserqrgulif2hQwqEel/GbOrqGqbYm+wwANhljNljP9kiYBzgWTNIBh6yl1cB7599uKouJ06cYPjw4TidTowxvPjiiy02ya9YsYK7776bmTNnnlOSV81QRTkc+qZaXT0bjFVKpF2ildD732El9Qv6QmS7wMYcQN5kiERgn8fjXGBgtW02AddjlXfGA21FpL0xpgCIFJEswAnMNcac8SEgIlOBqQCdO3du8C/RksTExNRa425pRo0axffffx/oMJS/GQOF33sk9Y2wfxOUnbTaI6Otni+Xzag8Y297fmBjbmJ8dSr4c+DPIjIZWA3kAfZHK12MMXkiciGwUkS2GGOq9NUzxrwEvARW6aamFzDGuHtnKOVLTW2WtRbvZEHVO0vzNsKpw1abIwIuSIOM2yvr6rHdgrKu7kveJPo8oJPH4yR7nZsxJh/rjB4RaQPcYIwptNvy7H/3iMhnQD+gaqfsekRGRlJQUED79u012SufMsZQUFDg7puvGlnpKevs3JXU8zfC0b12o0BCL+gxurKufl5yUNXVG4s3iX490F1EumEl+AnAJM8NRCQeOGKMqQB+idUDBxGJBU4ZY0rsbYYAf2hokElJSeTm5nLo0KGG7qpUvSIjI0lKSgp0GMGv3AmHtnucqX8FBz3q6tGdrISe+VO7rp5uDSGgzlm9id4Y4xSR6cAKrO6VC4wx20RkDpBljFkCXAn8XkQMVunGNTpXb+BFEanA6rM/17O3jrfCwsLo1q1bQ3dTSgWKMdaZuav0kr8R8r8Gp30ndmSMlcx7jrFvQsqAth0CGnIwq7d7ZWOrqXulUqqJO3HIo65u/3v6iNUWGmmdnbuGC0jMgLgLQcuwPnWu3SuVUqpSyYnKuroruRfavZ8kBBJ6Q68fVa2rO8ICG3MLp4leKVW78jKrju4+U99o1dmNPXZRTGfrLP2Suz3q6l7fH6kaiSZ6pZTFGDiyp7KmnrfBOnN32gPLRcVZZ+m9x1aWYdokBDZm5RVN9Eq1VCcOVq2p522AYmvuAkKjrLPzzDsrSzCxXbWu3kxpoleqJSg5bvV68bxgWmTf8C4hVh09+brKm5ASeoND00Ow0L+kUsHGWQoHt1XW1PM2WOPCYPewi+kCSZfAwHvtunoahLeu8ylV86aJXqnmrKLCqqt7DhmwfzOUl1jtrdpbybzPjysno24dH9iYVaPTRK9Uc3L8h6o19fyNUFxktYW1skZpHHB35eBeMV20rq400SvVZBUfg/1fVx3c65g9zJQ4oEMy9Blf2QMmoZfW1VWN9F2hVFPgLIUDW6v2gjm8E3ddPbYbdB5c2QPm/DQIbxXQkFXzoYleqcZWUQFHdledNOOHLVBearW3ireSecoNlSWYVnGBjVk1a5rolfK3Y/urJvX8r6DkmNUW1ho69oWB91R2bYzupHV15VOa6JXypeIiK5F7lmCO77faQkKt/uruM/X+kNATQhyBjVkFPU30Sp0tZwn8sLXq2XpBTmV73IXQ9bLKpH5+KoRFBS5e1WJpolfKW4X7YO8aj7r6Vqgos9pan2cl87Rb7PHV+2ldXTUZmuiVqk/JCVj9JPxnnpXYw9tYiXzwzyrP1tslal1dNVma6JWqjTGQ/Q9Y8Sur/3rfW2HwdK2rq2ZHE71SNTmcA0tnwp5V0CEVblwAnQcFOiqlzoomeqU8lZ60yjRr/2wNKTDmSWuyar3jVDVj+u5VCjzKNLPhWC6kT4IRj0Ob8wIdmVLnTBO9UmeUaV7VMo0KKproVctVehJWPwVrn7f6t4/5gzWjkpZpVJDRd7RqeYyB7Utg+a+0TKNaBE30qmU5vAuWzYTdK6FDCtzwCnQZHOiolPIrTfSqZSg9CWv+CP/+k1WmGf0/cMldWqZRLYK+y1VwMwa2fwDLf2mXaSbC1Y9D2w6BjkypRhPizUYiMlpEdojILhGZVUN7FxH5p4hsFpHPRCTJo+0OEcmxf+7wZfBK1enwLvi/G+Dt2yEqBqYsh/EvaJJXLU69Z/Qi4gDmASOAXGC9iCwxxmR7bPYU8IYx5nURuQr4PXC7iMQBjwKZWFPlbLD3PerrX0QpN1eZZu3zEBqpZRrV4nnzzh8A7DLG7AEQkUXAOMAz0ScDD9nLq4D37eVRwCfGmCP2vp8Ao4GF5x66UtUYA998aJVpivZB2gQYMUfP4FWL503pJhHY5/E4117naRNwvb08HmgrIu293BcRmSoiWSKSdejQIW9jV6pSwW6rTPO32yCiHUxZBte/qEleKbys0Xvh58BQEfkKGArkAeXe7myMeckYk2mMyUxISPBRSKpFKD0F/3wC/jIIctfD6Llwz2rocmmgI1OqyfCmdJMHdPJ4nGSvczPG5GOf0YtIG+AGY0yhiOQBV1bb97NziFcpi5ZplPKaN4l+PdBdRLphJfgJwCTPDUQkHjhijKkAfgkssJtWAL8TkVj78Ui7XamzV7Ablv0Cdn0K5/WByUuh65BAR6VUk1VvojfGOEVkOlbSdgALjDHbRGQOkGWMWYJ11v57ETHAauA+e98jIvIE1ocFwBzXhVmlGqz0lN2b5k/giLDKNJfcrb1plKqHGGMCHUMVmZmZJisrK9BhqKbEGPjmI7tM8701L+uIOdD2/EBHplSTISIbjDGZNbXpqZBq2gp2w7KHYdcncF6ylmmUOgua6FXTVHoKPn8a/v2cVaYZ9XsYcDc4wgIdmVLNjiZ61bQYAzuWwrJZVpkm9WYY+YSWaZQ6B5roVdNRsBuWz4KcjyGhN0z+CLpeFuiolGr2NNGrwCs9BZ8/A/9+1i7T/A4GTNUyjVI+ooleBY6WaZRqFJroVWAc2WP1ptEyjVJ+p4leNa6y01aZ5vNnwREOI38LA+/RMo1SfqSJXjUOY2DHMlj+MBR+D6k3wYgnoN0FgY5MqaCniV7535E9Vh0+ZwUk9II7PoRulwc6KqVaDE30yn+qlGnCtEyjVIBoolf+sWOZdbG18DtIuRFG/kbLNEoFiCZ65VtHvrVuetq5XMs0SjURmuiVb5Sdtko0nz9jl2l+AwPv1TKNUk2AJnp17nYstyYCcZdpnoB2HQMdlVLKpolenT3PMk18T7jjA+h2RaOGYIyhxFlBcVk5xWXWv6fLyt3/lpRVuB8XV1mubV3V53A9Li4rp9hZQYe2EaQmRZOWFENqYjSpidHEtg5v1N9ZqYbSRK8aruy0NXzwmqet0syIJ2DQNHeZpqLCSr6eCbd60jxdPYlWW3dmki7ndFkFJdWfz1nO2cydIwJRYQ4iwxxEhoYQGe4gMtRBVLiDyLAQoqPCPNaFEBnqICIshH1HTrMlr4gV2w64n6tTXBRpSTGkJUaTmhRNSmI07SK1ZKWaDk30LUR5hannbNa7s92LCz/npkPPc57zB1ZHDOXFiCnkrY3h9L/+5d6+1FlxVjE6QsROviFWAraXo8IctI4IpX2bysRsJeTKbTyTdFSYg4gwxxnrIu11keEhhDtCEJGzPp5Fp8vYllfEptwituQVsmlfIR9t3u9uvzChtZ34Y0hLiqZPx3a0Ctf/biowdCrBACorr0yknmevp0utMsHp0nJKnGeuK3ba29vLrrbazo6Ly8opKz+7v3O4I4SIsBAuDD3ML8xrDClfzz5HJxZET2d3mwyi7KTsSqQRHkn1zKRddV1Ute3DHCE+PsKN68jJUrbkFbF5XyGb84rYklvED8eKAQgRuPi8NtaZf5JV8ul9QTsiwxwBjloFi7qmEtRE78EYQ2l5BcWlFR4J1D6zdS3Xuq6i9iRdVrl9ibPyTLm84uyOfURoSC2J1OPMtcZ1lWe71ddVOTv2aHeUF1tlms+fAXHAlbOs3jShWpf2xsFjxVbyzy1ic24hm3OLKDhZCkBoiNCjQ1vSO0WTmmh9APTo0Jbw0Ob9gacCo0Uk+lOlTv6+Mc86O3ad+Xqe2XokaautokqSdiXtsz0cUdWSppVMz+5s1116OGN7BxGhIYSEnH3JoUF2rrB60xzdC32ut7pMRic2zmsHKWMM+4uK2WyXfKwPgCKKTpcB1jeo3he0tS72JkWTlhTNxQltCG3m33aU/7WIRF9wooT+v/nU/TjE82KbK2naF9d8dbbrKj1EhJ5bvbfJOboXlv/SGis+vidc8we48MoABxW8jDHsO3KazXmFbMktYlNuIVvzjnGixAlAZFgIfTpaSd8q+8RwYXzrxvvAV81Ci0j0FRWGwydK3D0lwhwSXMm3MZS5yjRP22Wah2HgNC3TBEBFheHbgpNssc/4t+RZyf90WTkAbSJC6dOxnZ38rbJP57hW+p5vwVpEolfnSMs0TZ6zvILdh06yObfQXffP3n/M3cspOiqM1ESPM/+kGDpGR2rybyE00avaVSnT9IBrntQyTTNS6qxg54Hj7sS/Ja+Qb/Yfx2lf6G/fOtyq9SdWnvmf1y4ywFErf6gr0WvH3paqrBjW/gnW/NEq04yYo2WaZig8NISUROsmrYkDrHXFZeV888Nxtti9fLbkFbF65yFcnbw6tItw9/JxdfVs3yYicL+E8juvEr2IjAaeAxzAK8aYudXaOwOvAzH2NrOMMUtFpCuwHdhhb7rOGHOvb0JXZ23nx3aZ5lvoM94aJ17LNEEjMsxB304x9O0U4153qtRJdv4xd+LfnFvIP7854O5llhgTZZd7oklPiiGlYzTRrfTu3mBRb6IXEQcwDxgB5ALrRWSJMSbbY7NHgLeNMfNFJBlYCnS123YbY/r6Nmx1Vo5+Z5dpPrLKNLe/DxcNC3RUqhG0Cg8ls2scmV3j3OuOF5exNe+Yu5vnlrwilm39wd3etX0r685ej6Ed2kRoEaA58uavNgDYZYzZAyAii4BxgGeiN0A7ezkayPdlkOoclRXD2udhzVNWmebqx2HQz7RM08K1jQxj8EXtGXxRe/e6wlOllfX+3CI2fneUDzZZ/51F4KKENu7En5YUTfIF0USF6929TZ03iT4R2OfxOBcYWG2bx4CPReR+oDVwtUdbNxH5CjgGPGKMWVP9BURkKjAVoHPnzl4Hr7yQ8wksnWmVaZJ/DKN+C/sthtQAABFwSURBVNFJgY5KNVExrcK5vHsCl3dPcK87fKKkSjfPNbsO8/ev8gBrfKLu57Vx9/JJS4ym1wVtiQjV5N+U+Op72ETgr8aYP4rIYOB/RSQF2A90NsYUiEh/4H0R6WOMOea5szHmJeAlsHrd+Cimlu3od7DiV/DNh9C+u5Zp1FmLbxPBsF7nMazXee51B44VVxnW4dPtB3k7KxeAMIfQ6/x27t4+qfbQDs19LKPmzJtEnwd08nicZK/zdCcwGsAY8x8RiQTijTEHgRJ7/QYR2Q30ALT/pL9UKdOEwNWPwaD7tEyjfKpDu0hGJEcyIrkDYN3dm1d42r6z1zrz/2BTPm998T1gjc+U3LFdlRE9L0pog0Pv7m0U3iT69UB3EemGleAnAJOqbfM9MBz4q4j0BiKBQyKSABwxxpSLyIVAd2CPz6JXVeV8CstmwpE9WqZRjUpESIptRVJsK8akWpPAV1QYvj9yis0eI3q+uyGX1//zHQCtwh2kdKys96cmRtO1vQ7t4A/1JnpjjFNEpgMrsLpOLjDGbBOROUCWMWYJ8N/AyyIyA+vC7GRjjBGRK4A5IlIGVAD3GmOO+O23aamqlGkuhtvfg4uuCnRUqoULCRG6xrema3xrrku3ppYsrzB8e/iEezC3zbmF/N+67yix7+5tGxlqzdyVFE2a3dc/KTZK7+49R3pnbHPmLLFuelr9R6tLxBUzYfB9EKo3v6jmw1leQc7BE9YFX7ur5/b9x9xzKMS2CqvSzTMtKZrz2+nQDtXpEAjBqEqZZhyM+p2WaVTQKHGWs/OHEx4jehax88Bx9xwO8W0i3Hf2ukb0TGjbsk9wdAiEYFL4vXXTk6tMc9vf4eLhgY5KKZ+KCHWQat+p6+rMXVxWTvb+Y+6unptzC1m146D77t4LoiPdo3nqxO1VaaJvLqqXaYY/qmUa1aJEhjnI6BxLRudY97qTJU625R9zj+i5JbeGidvtWn9LnrhdE31zsOtTWPoLOLIbel9nlWliOtW/n1JBrnVEKAO6xTGgW+XQDq6J213z9m7OK+SjLR4Tt8e3tmv9LWfi9uD+7Zq7wn2w4pew/QMt0yjlpeioMC69OJ5LL453r3NN3O4a0fPLb4/wj6+toR1cE7enJsbY8/cG38TtejG2KXKWWDc9rX7Kejx0JgyermUapXzIc+J214ieh09Unbjdcwavpj5xu/a6aU60TKNUQNQ0cfuWvCIKT1WduN3dx79T05q4XRN9c1C4z7rpafsSiLvImpD74qvr308p5TfGGHKPnmZTbqG7t8/WvCKOV5u4vXIKx8BN3K6JvilzlsB//myVaYyBK34Ol96vZRqlmqiKCsPegpPuu3vrmrg9NSmG9EaauF0TfVO165/WTE8Fu6D3tXaZRodpVqq5Ka8w7D50gk37ap64vV1kqNW/3zV/byffT9yuN0w1NdXLNLct1jKNUs2Yw75426NDW27KtK6plZXbE7d7jOj58uo9NU7c7jrz99fE7ZroG5Oz1C7TPGmVaa76f1qmUSpIhTms+n2fjtFM8Ji4fccPx90jelafuP2SrrG8c++lPo9FE31j8SzT9BoLo3+vZRqlWpjIMAfpnWJI7xQDg7oAcLq0nOz9Vrkn1E8XcTXR+1tRrlWmyf4HxF0Ity6G7lqmUUpZosId9O8SR/8ucfVvfJY00fvLGWWaR+DSB7RMo5RqdJro/WH3KmtC7oIcLdMopQJOE70vFeXCitmQ/b5dpnkXuo8IdFRKqRZOE70vOEth3Tz41x8qyzSD74cw/3SVUkqphtBEf66ql2lG/Q5iuwQ6KqWUctNEf7aK8uzeNO9DbDeY9A70GBnoqJRS6gya6BvKWQrr/mKXacph2CPWTU9aplFKNVGa6Btiz2dWmebwTuj5I6s3jZZplFJNnCZ6bxTlwcezYdt7WqZRSjU7mujrckaZZrZ105OWaZRSzYgm+tqcUab5HcR2DXRUSinVYJroqzuWb930tO3vVmKf9Db0GBXoqJRS6qx5NdmhiIwWkR0isktEZtXQ3llEVonIVyKyWUSu8Wj7pb3fDhFpuhnTWQr/fg6ez4QdS+HKX8HPvtAkr5Rq9uo9oxcRBzAPGAHkAutFZIkxJttjs0eAt40x80UkGVgKdLWXJwB9gI7ApyLSwxhT7utf5Jzs+ZddptkBPa+xe9N0DXRUSinlE96UbgYAu4wxewBEZBEwDvBM9AZoZy9HA/n28jhgkTGmBPhWRHbZz/cfH8R+7qqXaSb+DXqODnRUSinlU94k+kRgn8fjXGBgtW0eAz4WkfuB1oBrwPVEYF21fROrv4CITAWmAnTu3AijPJaXwbr58K//gQqnVaYZ8l/am0YpFZR8dTF2IvBXY8wfRWQw8L8ikuLtzsaYl4CXwJoc3Ecx1ezb1VaZ5tA30GOMVaaJ6+bXl1RKqUDyJtHnAZ08HifZ6zzdCYwGMMb8R0QigXgv920cx/Lh40dg62KI6aJlGqVUi+FNr5v1QHcR6SYi4VgXV5dU2+Z7YDiAiPQGIoFD9nYTRCRCRLoB3YEvfRW8V8rLYO3z8OdLYPuHcOUv4b4vNMkrpVqMes/ojTFOEZkOrAAcwAJjzDYRmQNkGWOWAP8NvCwiM7AuzE42xhhgm4i8jXXh1gnc16g9brRMo5RSiJWPm47MzEyTlZV1bk9ybL9dpnnXKtOM+R/oOcY3ASqlVBMkIhuMMZk1tQXXnbHlZfDFC/DZXGt56Cy47EEIiwp0ZEopFTDBk+iP7oW3brHLNKPtMs2FgY5KKaUCLngSfduOEJ0EVz+mZRqllPIQPIk+NBxuWxzoKJRSqsnxalAzpZRSzZcmeqWUCnKa6JVSKshpoldKqSCniV4ppYKcJnqllApymuiVUirIaaJXSqkgp4leKaWCnCZ6pZQKcprolVIqyGmiV0qpIKeJXimlgpwmeqWUCnKa6JVSKshpoldKqSCniV4ppYKcJnqllApymuiVUirIaaJXSqkgp4leKaWCnCZ6pZQKcl4lehEZLSI7RGSXiMyqof0ZEfna/tkpIoUebeUebUt8GbxSSqn6hda3gYg4gHnACCAXWC8iS4wx2a5tjDEzPLa/H+jn8RSnjTF9fReyUkqphvDmjH4AsMsYs8cYUwosAsbVsf1EYKEvglNKKXXuvEn0icA+j8e59roziEgXoBuw0mN1pIhkicg6EflxLftNtbfJOnTokJehK6WU8oavL8ZOAN41xpR7rOtijMkEJgHPishF1XcyxrxkjMk0xmQmJCT4OCSllGrZvEn0eUAnj8dJ9rqaTKBa2cYYk2f/uwf4jKr1e6WUUn7mTaJfD3QXkW4iEo6VzM/oPSMivYBY4D8e62JFJMJejgeGANnV91VKKeU/9fa6McY4RWQ6sAJwAAuMMdtEZA6QZYxxJf0JwCJjjPHYvTfwoohUYH2ozPXsraOUUsr/pGpeDrzMzEyTlZUV6DCUUqpZEZEN9vXQM+idsUopFeQ00SulVJDTRK+UUkFOE71SSgU5TfRKKRXkNNErpVSQ00SvlFJBThO9UkoFOU30SikV5DTRK6VUkNNEr5RSQU4TvVJKBTlN9EopFeQ00SulVJDTRK+UUkFOE71SSgU5TfRKKRXkNNErpVSQ00SvlFJBThO9UkoFOU30SikV5DTRK6VUkNNEr5RSQU4TvVJKBTlN9EopFeQ00SulVJDzKtGLyGgR2SEiu0RkVg3tz4jI1/bPThEp9Gi7Q0Ry7J87fBm8Ukqp+oXWt4GIOIB5wAggF1gvIkuMMdmubYwxMzy2vx/oZy/HAY8CmYABNtj7HvXpb6GUUqpW3pzRDwB2GWP2GGNKgUXAuDq2nwgstJdHAZ8YY47Yyf0TYPS5BKyUUqphvEn0icA+j8e59roziEgXoBuwsiH7ishUEckSkaxDhw55E7dSSikv+fpi7ATgXWNMeUN2Msa8ZIzJNMZkJiQk+DgkpZRq2bxJ9HlAJ4/HSfa6mkygsmzT0H2VUkr5gTeJfj3QXUS6iUg4VjJfUn0jEekFxAL/8Vi9AhgpIrEiEguMtNcppZRqJPX2ujHGOEVkOlaCdgALjDHbRGQOkGWMcSX9CcAiY4zx2PeIiDyB9WEBMMcYc8S3v4JSSqm6iEdebhIyMzNNVlZWoMNQSqlmRUQ2GGMya2rTO2OVUirIaaJXSqkg1+RKNyJyCPjuHJ4iHjjso3B8SeNqGI2rYTSuhgnGuLoYY2rsn97kEv25EpGs2upUgaRxNYzG1TAaV8O0tLi0dKOUUkFOE71SSgW5YEz0LwU6gFpoXA2jcTWMxtUwLSquoKvRK6WUqioYz+iVUkp50ESvlFJBrtkkei+mM4wQkb/Z7V+ISFePtl/a63eIyKhGjushEckWkc0i8k97zH5XW7nHFIxnDBTn57gmi8ghj9e/y6PNb9M/nuO0lP48XgtE5KCIbK2lXUTkT3bcm0Ukw6PNn8ervrhutePZIiJrRSTdo22vvf5rEfHpuCJexHWliBR5/L1+7dFW53vAz3HN9Ihpq/2eirPb/Hm8OonIKjsXbBOR/6phG/+9x4wxTf4HazC13cCFQDiwCUiuts3PgBfs5QnA3+zlZHv7CKxJUXYDjkaMaxjQyl6e5orLfnwigMdrMvDnGvaNA/bY/8bay7GNFVe17e/HGkTPr8fLfu4rgAxgay3t1wDLAAEGAV/4+3h5GdelrtcDxrjish/vBeIDdLyuBD481/eAr+Oqtu21wMpGOl4XABn2cltgZw3/J/32HmsuZ/TeTGc4DnjdXn4XGC4iYq9fZIwpMcZ8C+yyn69R4jLGrDLGnLIfrsMak9/fGjr9oyd/Tv94LtNS+pUxZjVQ18iq44A3jGUdECMiF+Dn6TLri8sYs9ZUzsHcWO8vb45Xbc7lvenruBrz/bXfGLPRXj4ObOfM2fb89h5rLonemykJ3dsYY5xAEdDey339GZenO7E+sV0ixZpCcZ2I/NhHMTUkrhvsr4jviohrgpgmcbzkzGkpwX/Hyxu1xe7P49VQ1d9fBvhYRDaIyNQAxDNYRDaJyDIR6WOvaxLHS0RaYSXLxR6rG+V4iVVW7gd8Ua3Jb++xesejV74hIrcBmcBQj9VdjDF5InIhsFJEthhjdjdSSB8AC40xJSJyD9a3oasa6bW9UdO0lIE8Xk2aiAzDSvSXeay+zD5e5wGfiMg39hlvY9iI9fc6ISLXAO8D3Rvptb1xLfBvU3V+DL8fLxFpg/Xh8qAx5pgvn7suzeWM3pspCd3biEgoEA0UeLmvP+NCRK4GZgPXGWNKXOuNMXn2v3uAz7A+5RslLmNMgUcsrwD9vd3Xn3F5qD4tpT+Plzdqiz3g02WKSBrW33CcMabAtd7jeB0E3sN3Jct6GWOOGWNO2MtLgTARiacJHC9bXe8vvxwvEQnDSvJvGmP+XsMm/nuP+ePCg69/sL557MH6Ku+6gNOn2jb3UfVi7Nv2ch+qXozdg+8uxnoTVz+si0/dq62PBSLs5XggBx9dlPIyrgs8lscD60zlhZ9v7fhi7eW4xorL3q4X1oUxaYzj5fEaXan94uKPqHqh7Et/Hy8v4+qMdd3p0mrrWwNtPZbXAqMbMa7zXX8/rIT5vX3svHoP+Csuuz0aq47furGOl/27vwE8W8c2fnuP+ezg+vsH64r0TqykOdteNwfrLBkgEnjHftN/CVzose9se78dwJhGjutT4ADwtf2zxF5/KbDFfqNvAe5s5Lh+D2yzX38V0Mtj35/ax3EXMKUx47IfPwbMrbafv4/XQmA/UIZVA70TuBe4124XYJ4d9xYgs5GOV31xvQIc9Xh/ZdnrL7SP1Sb77zy7keOa7vH+WofHB1FN74HGisveZjJWBw3P/fx9vC7Dugaw2eNvdU1jvcd0CASllApyzaVGr5RS6ixpoldKqSCniV4ppYKcJnqllApymuiVUirIaaJXSqkgp4leKaWC3P8HSGN7vFd4pmcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_history(lstm_history_pre)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That definitely helps the model learn faster, although performance still isn't amazing. A few takeaways: 1) LSTMs alleviate the challenges with training recurrent models, but don't entirely solve them, and 2) RNNs are not a silver bullet, and won't necessarily outperform other models such as CNNs.\n",
    "\n",
    "(Those things said, there are certainly many things that we could do to improve this model.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "* Evaluated vanilla RNN and LSTM models for binary text classification using the IMDB dataset\n",
    "* Observed failure of vanilla RNN to learn the task. [RNN training is difficult](http://proceedings.mlr.press/v28/pascanu13.pdf); vanishing/exploding gradients\n",
    "* Observed that replacing `SimpleRNN` with `LSTM` allowed the network to learn; LSTMs alleviate the training difficulties seen with vanilla RNNs."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}