seq2seq_dates.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sequence-to-sequence RNN\n",
    "\n",
    "This notebook shows an example of a character-level sequence-to-sequence recurrent neural network model using Keras.\n",
    "\n",
    "The implementation draws on [A ten-minute introduction to sequence-to-sequence learning in Keras](https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html) and [English to Katakana using a Sequence-to-Sequence model](https://github.com/wanasit/katakana)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration\n",
    "\n",
    "The following variables configure a few aspects of the data, model, and training process. To adapt this example to a different dataset, you'll probably want to change these to match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Maximum number of examples to read\n",
    "MAX_EXAMPLES = 100000\n",
    "\n",
    "# Maximum length of input sequence in characters\n",
    "INPUT_LENGTH = 35\n",
    "\n",
    "# Maximum length of output sequence in characters, including start symbol\n",
    "OUTPUT_LENGTH = 11\n",
    "\n",
    "# Number of epochs to train for\n",
    "EPOCHS = 3\n",
    "\n",
    "# Training batch size\n",
    "BATCH_SIZE = 64\n",
    "\n",
    "# Size of character embeddings\n",
    "EMBEDDING_DIM = 100\n",
    "\n",
    "# Size of RNN states\n",
    "RNN_UNITS = 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download dataset\n",
    "\n",
    "We'll be using an automatically generated dataset of freeform and standardized dates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File ‘generated_dates.txt’ already there; not retrieving.\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!wget -nc https://raw.githubusercontent.com/TurkuNLP/Deep_Learning_in_LangTech_course/master/data/generated_dates.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "max input length 35\n",
      "max output length 10\n",
      "ELOKUUN 23. 1977     →     23.08.1977\n",
      "23.10.1972     →     23.10.1972\n",
      "2006/12/22     →     22.12.2006\n",
      "1990/11/28     →     28.11.1990\n",
      "syyskuun 5. 2015     →     05.09.2015\n",
      "30/10/1975     →     30.10.1975\n",
      "Kesäkuun 28. vuonna 2001     →     28.06.2001\n",
      "2018/10/21     →     21.10.2018\n",
      "1996/4/6     →     06.04.1996\n",
      "1981/06/14     →     14.06.1981\n",
      "21/05/2017     →     21.05.2017\n",
      "18/01/1975     →     18.01.1975\n",
      "1/9/1970     →     01.09.1970\n",
      "Huhtikuun 24. päivänä 2011     →     24.04.2011\n",
      "6. päivänä Heinäkuuta 1970     →     06.07.1970\n",
      "maaliskuun 24. päivänä vuonna 1985     →     24.03.1985\n",
      "ELOKUUN 23. päivänä vuonna 2012     →     23.08.2012\n",
      "kesäkuun 8. päivänä 2017     →     08.06.2017\n",
      "2000.11.02     →     02.11.2000\n",
      "9. päivänä marraskuuta 1974     →     09.11.1974\n",
      "1979.08.15     →     15.08.1979\n",
      "24/08/1997     →     24.08.1997\n",
      "elokuun 23. päivänä 2014     →     23.08.2014\n",
      "19/9/1987     →     19.09.1987\n",
      "6/11/2015     →     06.11.2015\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "\n",
    "def load_data(fn, separator='\\t', has_header_line=False, max_examples=None):\n",
    "    data = []\n",
    "    with open(fn) as f:\n",
    "        if has_header_line:\n",
    "            next(f)    # skip header\n",
    "        for line in f:\n",
    "            if max_examples is not None and len(data) >= max_examples:\n",
    "                break\n",
    "            line = line.rstrip('\\n')\n",
    "            input_text, output_text = line.split(separator)\n",
    "            data.append([input_text, output_text])\n",
    "    return data\n",
    "\n",
    "\n",
    "data = load_data('generated_dates.txt', max_examples=MAX_EXAMPLES)\n",
    "\n",
    "random.seed(1234)    # make random.shuffle() repeatable\n",
    "random.shuffle(data)\n",
    "\n",
    "input_texts = [input_text for input_text, output_text in data]\n",
    "output_texts = [output_text for input_text, output_text in data]\n",
    "\n",
    "\n",
    "# Have a look at the source data\n",
    "print('max input length', max(len(t) for t in input_texts))\n",
    "print('max output length', max(len(t) for t in output_texts))\n",
    "for i in range(25):\n",
    "    print(input_texts[i], '    →    ', output_texts[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vectorize text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tokenize on the character level with lowercasing. To allow different input and output alphabets, create separate tokenizers for each.\n",
    "\n",
    "We're slightly abusing the tokenizer's out-of-vocabulary item support here to introduce a special `<START>` token into the mapping. As we're not restricting the number of words, this token will never be output by the tokenizer, so we're free to use it for our purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "\n",
    "\n",
    "input_tokenizer = Tokenizer(lower=True, char_level=True)\n",
    "output_tokenizer = Tokenizer(lower=True, char_level=True, oov_token='<START>')\n",
    "\n",
    "input_tokenizer.fit_on_texts(input_texts)\n",
    "output_tokenizer.fit_on_texts(output_texts)\n",
    "\n",
    "# Remember these\n",
    "INPUT_VOCAB_SIZE = max(input_tokenizer.word_index.values()) + 1\n",
    "OUTPUT_VOCAB_SIZE = max(output_tokenizer.word_index.values()) + 1\n",
    "START_INDEX = output_tokenizer.word_index['<START>']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at those mappings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries in input mapping: 31\n",
      "{' ': 2,\n",
      " '.': 8,\n",
      " '/': 9,\n",
      " '0': 4,\n",
      " '1': 1,\n",
      " '2': 6,\n",
      " '9': 5,\n",
      " 'n': 7,\n",
      " 'u': 3,\n",
      " 'ä': 10}\n",
      "Number of entries in output mapping: 12\n",
      "{'.': 2,\n",
      " '0': 3,\n",
      " '1': 4,\n",
      " '2': 5,\n",
      " '3': 9,\n",
      " '5': 10,\n",
      " '7': 8,\n",
      " '8': 7,\n",
      " '9': 6,\n",
      " '<START>': 1}\n"
     ]
    }
   ],
   "source": [
    "from pprint import pprint    # pretty-printer\n",
    "\n",
    "\n",
    "def truncate_dict(d, count=10):\n",
    "    # Returns at most count items from the given dictionary.  \n",
    "    return dict(i for i, _ in zip(d.items(), range(count)))\n",
    "\n",
    "\n",
    "print('Number of entries in input mapping:', len(input_tokenizer.word_index))\n",
    "pprint(truncate_dict(input_tokenizer.word_index))\n",
    "print('Number of entries in output mapping:', len(output_tokenizer.word_index))\n",
    "pprint(truncate_dict(output_tokenizer.word_index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Map input and output texts to integer sequences using the tokenizers, then pad and truncate the sequences to desired input and output lengths."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text: ELOKUUN 23. 1977\n",
      "Sequence: [27, 25, 16, 11, 3, 3, 7, 2, 6, 18, 8, 2, 1, 5, 13, 13]\n",
      "Padded:\n",
      "[27 25 16 11  3  3  7  2  6 18  8  2  1  5 13 13  0  0  0  0  0  0  0  0\n",
      "  0  0  0  0  0  0  0  0  0  0  0]\n",
      "Mapped back: ['e', 'l', 'o', 'k', 'u', 'u', 'n', ' ', '2', '3', '.', ' ', '1', '9', '7', '7', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-']\n",
      "Text: 23.08.1977\n",
      "Sequence: [5, 9, 2, 3, 7, 2, 4, 6, 8, 8]\n",
      "Padded:\n",
      "[5 9 2 3 7 2 4 6 8 8 0]\n",
      "Mapped back: ['2', '3', '.', '0', '8', '.', '1', '9', '7', '7', '-']\n",
      "type(encoder_X): <class 'numpy.ndarray'>\n",
      "encoder_X.shape: (100000, 35)\n",
      "decoder_Y.shape: (100000, 11)\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "\n",
    "def vectorize(texts, tokenizer, maxlen, quiet=False):\n",
    "    # This bit does the work\n",
    "    sequences = tokenizer.texts_to_sequences(texts)\n",
    "    padded = pad_sequences(sequences, maxlen=maxlen, padding='post')\n",
    "    \n",
    "    # This just prints out the first input and its vectorized versions\n",
    "    if not quiet:\n",
    "        print('Text:', texts[0])\n",
    "        print('Sequence:', sequences[0])\n",
    "        print('Padded:')\n",
    "        print(padded[0])\n",
    "        print('Mapped back:', [tokenizer.index_word.get(i, '-') for i in padded[0]])\n",
    "    \n",
    "    return padded\n",
    "\n",
    "\n",
    "encoder_X = vectorize(input_texts, input_tokenizer, INPUT_LENGTH)\n",
    "decoder_Y = vectorize(output_texts, output_tokenizer, OUTPUT_LENGTH)\n",
    "\n",
    "# This creates numpy arrays:\n",
    "print('type(encoder_X):', type(encoder_X))\n",
    "print('encoder_X.shape:', encoder_X.shape)\n",
    "print('decoder_Y.shape:', decoder_Y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In prediction, the decoder will receive its last output as input at each timestep. During training, we'll use _teacher forcing_, where the decoder is instead given the correct previous output. To implement this, the output sequence (`decoder_Y`) is shifted one character forward, and the special start symbol is placed first in the sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "decoder_Y: [5 9 2 3 7 2 4 6 8 8 0]\n",
      "Mapped back: ['2', '3', '.', '0', '8', '.', '1', '9', '7', '7', '-']\n",
      "decoder_X: [1 5 9 2 3 7 2 4 6 8 8]\n",
      "Mapped back: ['<START>', '2', '3', '.', '0', '8', '.', '1', '9', '7', '7']\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "\n",
    "decoder_X = np.zeros_like(decoder_Y)\n",
    "decoder_X[:,1:] = decoder_Y[:,:-1]\n",
    "decoder_X[:,0] = START_INDEX\n",
    "\n",
    "print('decoder_Y:', decoder_Y[0])\n",
    "print('Mapped back:', [output_tokenizer.index_word.get(i, '-') for i in decoder_Y[0]])\n",
    "print('decoder_X:', decoder_X[0])\n",
    "print('Mapped back:', [output_tokenizer.index_word.get(i, '-') for i in decoder_X[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build model\n",
    "\n",
    "Note the following:\n",
    "\n",
    "* `mask_zero=True` for the embedding makes the model ignore padding (see [Masking and padding with Keras](https://www.tensorflow.org/guide/keras/masking_and_padding))\n",
    "* `return_state=True` for the encoder RNN returns the state of the last timestep in addition to output (see https://keras.io/layers/recurrent/).\n",
    "* As we're using an LSTM, we get two separate state values, _h_ and _c_ (see below)\n",
    "* `return_sequences=True` for the decoder RNN returns the output from each time step, not only the last\n",
    "\n",
    "<img src=\"https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-chain.png\" style=\"width: 50%\">\n",
    "\n",
    "(LSTM illustration from https://colah.github.io/posts/2015-08-Understanding-LSTMs/ )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/smp/Library/Python/3.7/lib/python/site-packages/tensorflow_core/python/keras/initializers.py:119: calling RandomUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
      "WARNING:tensorflow:From /Users/smp/Library/Python/3.7/lib/python/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n",
      "WARNING:tensorflow:From /Users/smp/Library/Python/3.7/lib/python/site-packages/tensorflow_core/python/keras/backend.py:3872: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            [(None, 35)]         0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_2 (InputLayer)            [(None, 11)]         0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding (Embedding)           (None, 35, 100)      3200        input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 11, 100)      1300        input_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "lstm (LSTM)                     [(None, 100), (None, 80400       embedding[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "lstm_1 (LSTM)                   (None, 11, 100)      80400       embedding_1[0][0]                \n",
      "                                                                 lstm[0][1]                       \n",
      "                                                                 lstm[0][2]                       \n",
      "__________________________________________________________________________________________________\n",
      "time_distributed (TimeDistribut (None, 11, 13)       1313        lstm_1[0][0]                     \n",
      "==================================================================================================\n",
      "Total params: 166,613\n",
      "Trainable params: 166,613\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.layers import Input, Embedding, LSTM, TimeDistributed, Dense\n",
    "from tensorflow.keras.models import Model, load_model\n",
    "\n",
    "\n",
    "def build_seq2seq_model(input_length, output_length,\n",
    "                        input_vocab_size, output_vocab_size,\n",
    "                        embedding_dim=EMBEDDING_DIM, rnn_units=RNN_UNITS):\n",
    "    encoder_input = Input(shape=(input_length,))\n",
    "    encoder_embedding = Embedding(input_vocab_size, embedding_dim, mask_zero=True)(encoder_input)\n",
    "    encoder_output, encoder_h, encoder_c = LSTM(\n",
    "        rnn_units,\n",
    "        return_sequences=False,\n",
    "        return_state=True,\n",
    "        unroll=True\n",
    "    )(encoder_embedding)\n",
    "    encoder_states = [encoder_h, encoder_c]\n",
    "    \n",
    "    decoder_input = Input(shape=(output_length,))\n",
    "    decoder_embedding = Embedding(output_vocab_size, embedding_dim, mask_zero=True)(decoder_input)\n",
    "    decoder_rnn = LSTM(rnn_units, return_sequences=True, unroll=True)(\n",
    "        decoder_embedding,\n",
    "        initial_state=encoder_states\n",
    "    )\n",
    "    decoder_output = TimeDistributed(Dense(output_vocab_size, activation=\"softmax\"))(decoder_rnn)\n",
    "\n",
    "    model = Model(inputs=[encoder_input, decoder_input], outputs=[decoder_output])\n",
    "    return model\n",
    "\n",
    "\n",
    "model = build_seq2seq_model(\n",
    "    input_length=INPUT_LENGTH,\n",
    "    output_length=OUTPUT_LENGTH,\n",
    "    input_vocab_size=INPUT_VOCAB_SIZE,\n",
    "    output_vocab_size=OUTPUT_VOCAB_SIZE\n",
    ")\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "As our output values here are (as usual) integer values standing for our vocabulary items (characters), we'll use `sparse_categorical_crossentropy` loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(\n",
    "    optimizer='adam',\n",
    "    loss='sparse_categorical_crossentropy'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 80000 samples, validate on 20000 samples\n",
      "Epoch 1/3\n",
      "80000/80000 [==============================] - 116s 1ms/sample - loss: 0.5556 - val_loss: 0.1366\n",
      "Epoch 2/3\n",
      "80000/80000 [==============================] - 133s 2ms/sample - loss: 0.0431 - val_loss: 0.0092\n",
      "Epoch 3/3\n",
      "80000/80000 [==============================] - 123s 2ms/sample - loss: 0.0062 - val_loss: 0.0019\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(\n",
    "    x=[encoder_X, decoder_X], \n",
    "    y=[decoder_Y],\n",
    "    batch_size=BATCH_SIZE,\n",
    "    epochs=EPOCHS,\n",
    "    validation_split=0.2\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the training history:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3dd3wUZf7A8c+TTYMkJKTQS0BqSCMs3cahglgQ4RREAQ/kJ+qh3Ily6qnHeR6Wn+X8IVLUAxtiRwGx4Vk4SijSSUIACWAgoaWQ/vz+mE2yCQnZJJvM7ub7fr32ld2Z2ZlvJpvvPPvMd55RWmuEEEK4Py+zAxBCCOEcktCFEMJDSEIXQggPIQldCCE8hCR0IYTwEN5mbTg8PFxHRkaatXkhhHBLW7ZsydBaR1Q1z7SEHhkZSWJiolmbF0IIt6SUOlzdPOlyEUIIDyEJXQghPIQkdCGE8BCm9aELIRxTWFhIWloaeXl5ZociGpG/vz8dOnTAx8fH4fdIQhfCxaWlpREUFERkZCRKKbPDEY1Aa01mZiZpaWl06dLF4fdJl4sQLi4vL4+wsDBJ5k2IUoqwsLBafyuThC6EG5Bk3vTU5W/udgn9YEYOz3y5Dxn2VwghKnK7hP7NnnQWfH+A+etSzA5FiCYhMzOT+Ph44uPjadOmDe3bty97XVBQ4NA67rzzTvbv33/RZebPn88777zjjJBr5bvvvmPDhg1VzluyZAkPPPBAI0dUd253UnTaZV3Yc/wcz3+VRLdWgYyMbmt2SEJ4tLCwMLZv3w7Ak08+SWBgIA8++GCFZbTWaK3x8qq6jfjmm2/WuJ177723/sHWwXfffUd4eDiDBg0yZfvO5HYtdKUU/7w5hr6dQpj1/i/sOnrW7JCEaJJSUlKIiopi4sSJ9OnTh+PHjzN9+nSsVit9+vRh7ty5ZcteeumlbN++naKiIkJCQpgzZw5xcXEMHjyYEydOAPDYY4/x0ksvlS0/Z84cBgwYQM+ePVm/fj0AOTk5jB07lqioKMaNG4fVai072NibPXs2UVFRxMbG8vDDDwOQnp7OzTffjNVqZcCAAWzYsIEDBw6wZMkSnnvuOeLj48u2U5WDBw8ybNgwYmNjufrqq0lLSwNg+fLlREdHExcXx7BhwwDYuXMn/fv3Jz4+ntjYWFJTU52wx2vmdi10AH8fCwvv6MdN//czdy1L5LP7htIqyN/ssIRocH/7fDd7jp1z6jqj2rXgiRv61Om9+/btY9myZVitVgDmzZtHaGgoRUVFDBs2jHHjxhEVFVXhPWfPnuWKK65g3rx5/OlPf+KNN95gzpw5F6xba82mTZtYuXIlc+fO5csvv+SVV16hTZs2fPTRR/zyyy8kJCRc8L709HRWr17N7t27UUpx5swZAGbOnMlDDz3EoEGDOHToENdffz27du1i2rRphIeH19i1cs899zBt2jQmTpzIokWLeOCBB/jwww/529/+xvfff0/r1q3LtvXqq6/y4IMPcuutt5Kfn99o5/zcroVeqlWQP4snWzmTW8j0ZVvIKyw2OyQhmpxLLrmkLJkDvPfeeyQkJJCQkMDevXvZs2fPBe9p1qwZ1157LQD9+vXj0KFDVa775ptvvmCZn376ifHjxwMQFxdHnz4XHohCQ0Px8vLirrvu4pNPPiEgIACAb775hrvvvpv4+HhuuukmTp8+zfnz5x3+XTdu3Fi27UmTJvHjjz8CMHToUCZNmsSSJUsoKSkBYMiQITz11FM8++yzHDlyBH//xmlwumULvVSfdsG8eGscd7+9lTkf7eDFW+OlvEt4tLq2pBtKabIESE5O5uWXX2bTpk2EhIRw++23V1lH7evrW/bcYrFQVFRU5br9/PxqXKYqPj4+JCYm8vXXX/PBBx+wYMECvvrqq7IWv/32nWHx4sVs3LiRL774goSEBLZt28Ydd9zB4MGDWbVqFSNHjuSNN97g8ssvd+p2q+K2LfRSI6Pb8uere/Dp9mO8+v0Bs8MRosk6d+4cQUFBtGjRguPHj7N27Vqnb2Po0KGsWLECMPqpq/oGkJWVxblz57j++ut58cUX2bZtGwBXXXUV8+fPL1uutO89KCiIrKysGrc9aNCgsm2//fbbZQk6NTWVQYMG8fe//52WLVty9OhRUlNT6datG/fffz/XX389O3bsqN8v7iC3T+gA9/2uGzfGteO5tftZu/s3s8MRoklKSEggKiqKXr16MWnSJIYOHer0bfzxj3/k6NGjREVF8be//Y2oqCiCg4MrLHP27Fmuu+464uLiuOKKK3jhhRcAoyzy559/JjY2lqioKBYvXgzA6NGjWbFiBX379r3oSdH58+ezaNEiYmNjef/993nxxRcBmDVrFjExMcTExDBs2DCio6N599136dOnD/Hx8SQlJXH77bc7fV9URZl1gY7VatXOvMFFXmExty78L8knsvnw7iFEtWvhtHULYaa9e/fSu3dvs8NwCUVFRRQVFeHv709ycjLXXHMNycnJeHu7de9xtar62yultmitrVUt7xEtdDAqXxZPstLC34dpSzdzMivf7JCEEE6WnZ3N0KFDiYuLY+zYsSxcuNBjk3ldeNSeaNXCnyWTrYx7bT3/81Yi7941CH8fi9lhCSGcJCQkhC1btpgdhsvymBZ6qej2wbxwSzxbfz3DIx/vlDFfhBBNhscldIBRMW2ZdVUPPt52lNf+0zhXaAkhhNk8qsvF3szh3Ug+kcWza/fRrVUgV0e1NjskIYRoUB7ZQgdjzJfnfx9HTPtg7l++jb3HnXu5tBBCuBqPTehQXvkS5O/NtKWJZGRL5YsQtTVs2LALLhJ66aWXmDFjxkXfFxgYCMCxY8cYN25clctceeWV1FS+/NJLL5Gbm1v2etSoUWVjpjSWQ4cO8e6771Y7Lzo6ulHjqY5HJ3SA1i38WTzJSmZOPne/tYX8IhnzRYjamDBhAsuXL68wbfny5UyYMMGh97dr144PP/ywztuvnNBXr15NSEhInddXFxdL6K7EoYSulBqplNqvlEpRSl0wLJpSaopS6qRSarvtMc35odZdbIcQnv99HImHT/PoJ7uk8kWIWhg3bhyrVq0qu5nFoUOHOHbsGJdddhnZ2dkMHz6chIQEYmJi+Oyzzy54v30L9vz584wfP57evXszZsyYCoNjzZgxo2zo3SeeeAKAf/3rXxw7doxhw4aVDU0bGRlJRkYGAC+88ALR0dFER0eXDb176NAhevfuzV133UWfPn245pprqhyE64MPPigb9rb0Mv7i4mJmz55N//79iY2NZeHChQDMmTOHH3/8kfj4+LIrRKuSl5fHnXfeSUxMDH379mXdunUA7N69mwEDBpQNp5ucnExOTk7ZFa3R0dG8//77tfirVK3Gk6JKKQswH7gaSAM2K6VWaq0rD6Lwvtb6vnpH1ECuj21Hcno2L3+bTI/WgUy//BKzQxKi9tbMgd92OnedbWLg2nnVzg4NDWXAgAGsWbOG0aNHs3z5cm655RaUUvj7+/PJJ5/QokULMjIyGDRoEDfeeGO1g+QtWLCA5s2bs3fvXnbs2FFh+Nt//OMfhIaGUlxczPDhw9mxYwczZ87khRdeYN26dYSHh1dY15YtW3jzzTfZuHEjWmsGDhzIFVdcQcuWLUlOTua9995j8eLF3HLLLXz00UcXXH4/d+5c1q5dS/v27cu6cF5//XWCg4PZvHkz+fn5DB06lGuuuYZ58+bx/PPP88UXX1x0V86fPx+lFDt37mTfvn1cc801JCUl8dprr3H//fczceJECgoKKC4uZvXq1bRr145Vq1YBxpAF9eVIC30AkKK1TtVaFwDLgdH13rIJ7h/eneti2vLPNfv4dm+62eEI4Tbsu13su1u01jzyyCPExsZy1VVXcfToUdLTq//f+uGHH8oSa2xsLLGxsWXzVqxYQUJCAn379mX37t1VDrxl76effmLMmDEEBAQQGBjIzTffXDakbZcuXYiPjweqH6J36NChTJkyhcWLF1NcbHTFfvXVVyxbtoz4+HgGDhxIZmYmycnJDu4lI6bS369Xr1507tyZpKQkBg8ezNNPP80zzzzD4cOHadasGTExMXz99dc8/PDD/PjjjxeMSVMXjpQttgeO2L1OAwZWsdxYpdTlQBIwS2t9pPICSqnpwHSATp061T7aevLyMipfDp/KYeZ72/j4nqH0bBPU6HEIUWcXaUk3pNGjRzNr1iy2bt1Kbm4u/fr1A+Cdd97h5MmTbNmyBR8fHyIjI6scMrcmBw8e5Pnnn2fz5s20bNmSKVOm1Gk9pUqH3gVj+N2qulxee+01Nm7cyKpVq+jXrx9btmxBa80rr7zCiBEjKiz7/fff1zkWgNtuu42BAweyatUqRo0axcKFC/nd737H1q1bWb16NY899hjDhw/n8ccfr9d2nHVS9HMgUmsdC3wNLK1qIa31Iq21VWttjYiIcNKma6eZr1H5EuDnzdSlm8mUyhchahQYGMiwYcP4wx/+UOFk6NmzZ2nVqhU+Pj6sW7eOw4cPX3Q9l19+ednJxV27dpUNK3vu3DkCAgIIDg4mPT2dNWvWlL2nuuFtL7vsMj799FNyc3PJycnhk08+4bLLLnP4dzpw4AADBw5k7ty5REREcOTIEUaMGMGCBQsoLCwEICkpiZycHIeH2L3sssvKbnSdlJTEr7/+Ss+ePUlNTaVr167MnDmT0aNHs2PHDo4dO0bz5s25/fbbmT17Nlu3bnU49uo40kI/CnS0e93BNq2M1jrT7uUS4Nl6R9aA2gY3Y9EkK7cu/C8z3t7K29MG4uvt8QU/QtTLhAkTGDNmTIWKl4kTJ3LDDTcQExOD1WqlV69eF13HjBkzuPPOO+nduze9e/cua+nHxcXRt29fevXqRceOHSsMvTt9+nRGjhxJu3btyk4ygjFc75QpUxgwYAAA06ZNo2/fvtXeAamy2bNnk5ycjNaa4cOHExcXR2xsLIcOHSIhIQGtNREREXz66afExsZisViIi4tjypQpzJo1q8p13nPPPcyYMYOYmBi8vb3597//jZ+fHytWrOCtt97Cx8eHNm3a8Mgjj7B582Zmz56Nl5cXPj4+LFiwwKG4L6bG4XOVUt4Y3SjDMRL5ZuA2rfVuu2Xaaq2P256PAR7WWl/0FtrOHj63Lj7bfpT7l2/nFmsHnhkbK3c7Ei5Jhs9tumo7fG6NLXStdZFS6j5gLWAB3tBa71ZKzQUStdYrgZlKqRuBIuAUMKV+v0bjGB3fnpQT2bzyXQo9Wgcx7bKuZockhBB15tBYLlrr1cDqStMet3v+F+Avzg2tccy6qgcpJ7J5evVeLokIZFivVmaHJIQQddLkO469vBT/e0scvdu24I/vbSMpveYTH0I0NrkYrumpy9+8ySd0gOa+3iyeZMXfx8LUpZs5lVNgdkhClPH39yczM1OSehOitSYzMxN/f/9avc9j7inqDNt+Pc2tizbQt2MIb02VyhfhGgoLC0lLS6tXXbZwP/7+/nTo0AEfH58K0+t1UrQp6dupJc+Ni+X+5dt5YuUunh4TI5UvwnQ+Pj506dLF7DCEG5CEXsno+PYkpWcxf90BurcK4g+Xyj+SEMI9SEKvwp+v7knKiWyeWrWHrhEBXNlTKl+EEK5POomr4OWleOGWeHq2acEf391GygmpfBFCuD5J6NUI8PNmyWQrfj4Wpi5N5LRUvgghXJwk9ItoH9KMhXf04/iZPGa8s4XC4hKzQxJCiGpJQq9Bv84tmTc2hg2pp3hi5W6pBRZCuCw5KeqAmxM6kJSezWv/OUCPVoFMGSqVL0II1yMtdAc9NKInV/Vuzdwv9vBD0kmzwxFCiAtIQneQl5fipfHx9GgdxL3vbiXlRLbZIQkhRAWS0Gsh0Fb54mvxYtrSzZzJlcoXIYTrkIReSx1aNmfhHf04diaPe97ZKpUvQgiXIQm9DqyRoTx9cwzrD2Tyt8931/wGIYRoBFLlUkfj+nUgOT2LhT+k0rN1EHcMjjQ7JCFEEyct9Hp4aGQvhvdqxZOf7+Gn5AyzwxFCNHGS0OvB4qV4eUJfukUEcs87W0g9KZUvQgjzSEKvp9LKF2+LF9OWJnI2t9DskIQQTZQkdCfoGGpUvhw5ncu9726lSCpfhBAmkITuJP0jQ/nHmBh+Ssng71/sMTscIUQTJFUuTnSLtSPJ6Vks/vEg3VsHcfugzmaHJIRoQqSF7mRzru3NsJ4RPLFyN+tTpPJFCNF4JKE7mcVL8a8JfekaHsCMd7ZyMCPH7JCEEE2EJPQGEOTvw+uT++OlYOrSzZw9L5UvQoiGJwm9gXQKa86C2/vxa2Yu90nlixCiETiU0JVSI5VS+5VSKUqpORdZbqxSSiulrM4L0X0N6hrGUzdF82NyBk+t2mt2OEIID1djlYtSygLMB64G0oDNSqmVWus9lZYLAu4HNjZEoO5q/IBOJKVn88bPB+nROojbBnYyOyQhhIdypIU+AEjRWqdqrQuA5cDoKpb7O/AMkOfE+DzCI6N6cUWPCB7/bBfrD0jlixCiYTiS0NsDR+xep9mmlVFKJQAdtdarLrYipdR0pVSiUirx5Mmmcxs3b4sXr9zWl8jwAO55ZyuHM6XyRQjhfPU+KaqU8gJeAP5c07Ja60Vaa6vW2hoREVHfTbuVFv4+vD7ZOLUwdWki5/Kk8kUI4VyOJPSjQEe71x1s00oFAdHA90qpQ8AgYKWcGL1Q57AAFkzsx6GMHGa+t43iEm12SEIID+JIQt8MdFdKdVFK+QLjgZWlM7XWZ7XW4VrrSK11JLABuFFrndggEbu5wZeEMXd0NN/vP8nTq6XyRQjhPDVWuWiti5RS9wFrAQvwhtZ6t1JqLpCotV558TWIym4b2Imk9Cxe/+kgPVoHcmt/qXwRQtSfQ4Nzaa1XA6srTXu8mmWvrH9Ynu+x63qTmpHDY5/uIjIsgIFdw8wOSQjh5uRKUZN4W7x4ZUJfOoY25+63t/BrZq7ZIQkh3JwkdBMFNzPGfCnRxpgvWVL5IoSoB0noJusSHsCCiQmkSuWLEKKeJKG7gCHdwnnyxj6s23+SeWuk8kUIUTdyxyIXccegzhXudnSLtWPNbxJCCDvSQnchj18fxaXdwnn0k51sOnjK7HCEEG5GEroL8bZ4Mf+2BDq2NCpfjpySyhchhOMkobuY4OY+LJlspai4hGlLE8nOLzI7JCGEm5CE7oK6RgTy6sR+pJzM5n6pfBFCOEgSuou6tHs4T9wQxbf7TvDs2n1mhyOEcANS5eLCJg2OJCk9i4X/SaV7qyDG9etgdkhCCBcmLXQX98QNfRjaLYxHPt7JlsNS+SKEqJ4kdBfnY6t8aRfiz/RlW0g7LZUvQoiqSUJ3AyHNfVkyuT8FtsqXHKl8EUJUQRK6m+jWKpD5tyWQfCKbB97fTolUvgghKpGE7kYu7xHBX6/rzdd70nnuq/1mhyOEcDFS5eJmJg+JZH96Ngu+P0D3VoHcnCCVL0IIg7TQ3YxSirmj+zCoayhzPtrJlsOnzQ5JCOEiJKG7IR+LFwsm9qNtiD//81YiR8+cNzskIYQLkITuploG+PL6ZCv5hVL5IoQwSEJ3Y91aBfHKbX3Z/9s5ZknlixBNniR0N3dlz1Y8el0UX+1J53+/lsoXIZoyqXLxAH8YGklyehbz1x2gR+sgRse3NzskIYQJpIXuAYzKl2gGdgll9oc72ParVL4I0RRJQvcQvt5eLLi9H21a+DP9rS0ck8oXIZocSegeJDTAlyWTrZwvKOauZYnkFkjlixBNiUMJXSk1Uim1XymVopSaU8X8u5VSO5VS25VSPymlopwfqnBEj9ZBvDKhL3uPn+PPK36RyhchmpAaE7pSygLMB64FooAJVSTsd7XWMVrreOBZ4AWnRyocNqxXKx4Z1Zs1u37jpW+SzA5HCNFIHGmhDwBStNapWusCYDkw2n4BrfU5u5cBgDQLTTb10i7cYu3Av75LYeUvx8wORwjRCBwpW2wPHLF7nQYMrLyQUupe4E+AL/C7qlaklJoOTAfo1KlTbWMVtaCU4qmbYjiUkcvsD36hU2hz4juGmB2WEKIBOe2kqNZ6vtb6EuBh4LFqllmktbZqra0RERHO2rSohlH5kkBEkB/TlyXy29k8s0MSQjQgRxL6UaCj3esOtmnVWQ7cVJ+ghPOEBfrx+uT+5OQXcdeyRM4XFJsdkhCigTiS0DcD3ZVSXZRSvsB4YKX9Akqp7nYvrwOSnReiqK+ebYL414S+7Dp2lgc/kMoXITxVjQlda10E3AesBfYCK7TWu5VSc5VSN9oWu08ptVsptR2jH31yg0Us6mR479bMGdmLVTuP8/K3crwVwhM5NJaL1no1sLrStMftnt/v5LhEA5h+eVeS0rN5+dtkurcO5PrYdmaHJIRwIrlStAlRSvH0zdFYO7fkzyt+YUfaGbNDEkI4kST0JsbP28Jrd/QjPNCPu5Ylkn5OKl+E8BSS0Jug8EA/lky2kpVXxPRlieQVSuWLEJ5AEnoT1bttC14e35cdR88y+8MdaC2VL0K4O0noTdjVUa15aEQvPv/lGK98l2J2OEKIepI7FjVxd1/RleQTWbzwdRLdWwVybUxbs0MSQtSRtNCbOKUUT4+JIaFTCLNWbGfX0bNmhySEqCNJ6AJ/HwsL77ASFuDHtKWJnJDKFyHckiR0AUBEkB+LJ1k5e76Qu97aIpUvQrghSeiiTFS7Frx4azy/HDnDQ1L5IoTbkYQuKhgZ3YbZI3qy8pdjzF8nlS9CuBOpchEXuOfKS0hOz+L5r5Lo1iqQkdFS+SKEO5AWuriAUop5Y2OJ7xjCrPd/kcoXIdyEJHRRJX8fC4sm9SOkuQ93LUvkRJZUvgjh6iShi2q1CvJn8SQrZ3ILmb5MKl+EcHWS0MVFRbcP5sVb49h+5Ax/+XinVL4I4cIkoYsajYxuy5+v7sEn246y4D8HzA5HCFENqXIRDrnvd91IPpHNc2v30y0ikGv6tDE7JCFEJdJCFw5RSvHsuFhi2wfzwPvb2XPsnNkhCSEqkYQuHObvY2HxJCst/I3Kl5NZ+WaHJISwIwld1EqrFv4smWwlMyefu9/eQn6RVL4I4SokoYtai24fzAu3xLPl8GmpfBHChUhCF3UyKqYts67qwcdbj7Lwh1SzwxFCIFUuoh5mDu9G8oksnvlyH5dEBHJ1VGuzQxKiSZMWuqgzpRTPjYsjpn0wDyzfxr7fpPJFCDNJQhf10szXwqI7rAT4eTP134lkZEvlixBmkYQu6q1NsDHmS0Z2Pne/JZUvQpjFoYSulBqplNqvlEpRSs2pYv6flFJ7lFI7lFLfKqU6Oz9U4criOobw/O/jSDx8mkc/2SWVL0KYoMaErpSyAPOBa4EoYIJSKqrSYtsAq9Y6FvgQeNbZgQrXd0NcO2YO786HW9JY/KNUvgjR2BxpoQ8AUrTWqVrrAmA5MNp+Aa31Oq11ru3lBqCDc8MU7uKB4d0ZFdOGf67Zx3f70s0OR4gmxZGE3h44Yvc6zTatOlOBNVXNUEpNV0olKqUST5486XiUwm14eSn+9/fx9GnXgpnvbScpPcvskIRoMpx6UlQpdTtgBZ6rar7WepHW2qq1tkZERDhz08KFNPM1xnxp7mth6tLNnMopMDskIZoERxL6UaCj3esOtmkVKKWuAh4FbtRaS+1aE9c2uBmLJlk5cc4Y86WgqMTskITweI4k9M1Ad6VUF6WULzAeWGm/gFKqL7AQI5mfcH6Ywh3Fdwzh2XGxbDp4ir9+KpUvQjS0Gi/911oXKaXuA9YCFuANrfVupdRcIFFrvRKjiyUQ+EApBfCr1vrGBoxbuInR8e1JOZHNK9+l0L11INMu62p2SEJ4LIfGctFarwZWV5r2uN3zq5wcl/Ags67qQcqJbJ5evZdLWgUyrGcrs0MSwiPJlaKiwXl5Kf73ljh6t23BzHe3kSyVL0I0CEnoolE09/Vm8SQrfj4Wpi5NlMoXIRqAJHTRaNqFNGPRpH78di6PGVL5IoTTSUIXjSqhU0ueHRvLxoOneGKlVL4I4UxygwvR6G7q256k9Cxe/f4A3VsF8YdLu5gdkhAeQVrowhQPXtOTa6Ja89SqPXy/Xy5dEMIZJKELU3h5KV68NZ6ebVrwx3e3kXJCKl+EqC9J6MI0AX7eLJlsxc/Hi6lLEzktlS9C1IskdGGq9iHNWHiHleNn8rj33a0UFkvlixB1JQldmK5f55bMGxvD+gOZPLlyt1S+CFFHUuUiXMLNCR1ISs/mtf8coEfrICYPiTQ7JCHcjrTQhct4aERPrurdmrlf7OHHZLkBihC1JQlduAwvL8VL4+Pp3iqQe97ZyoGT2WaHJIRbkYQuXEqgrfLF1+LFtKWJnMmVyhchHCUJXbicDi2bs/COfhw9fV4qX4SoBfdL6MVFIFUQHs8aGco/xkTzc0omcz/fY3Y4QrgF90vo29+BF6Phiz9B8tdQmGd2RKKB/N7akemXd+WtDYd567+HzA5HCJfnfmWLIZ2gXTz88h4kvg4+zaHrMOgxwngEtTE7QuFED4/sxYET2Tz5+R66hAdyafdws0MSwmUpsy7isFqtOjExse4rKMyDQz9B0hrY/yWcSzOmt+sLPa41knvbODDucSrcWFZeIWMXrOe3s3l8eu9QukYEmh2SEKZRSm3RWlurnOe2Cd2e1pC+G5K+NB5piYCGoHa2lvtI6HI5+DZ3zvZEoztyKpfR838mpJkPn9wzlODmPmaHJIQpPD+hV5Z9EpK/MpL7ge+gIBu8m0HXK4zk3mMEtGjXMNsWDWbTwVNMXLKBgV3C+Ped/fG2uN8pICHqq+kldHtF+XD4Z6NbJmkNnPnVmN42zpbcR0LbePCS5OAOVmw+wkMf7WDKkEievLGP2eEI0eiadkK3pzWc3Ge03Pd/CWmbQJdAYOvyrpmuV4JvQOPGJWrlqS/2sOSng/xjTDQTB3Y2OxwhGpUk9OrkZELK10aCT/kW8s+Bxc/ob+85ErqPgJCO5sYoLlBcopm2dDM/JmewbOoAhlwilS+i6ZCE7oiiAvj1v7bW+xo4fdCY3jrGaL33vBbaJUjXjIvIyivk5lfXcyIrn8/uHUpkuHyrEk2DJPTa0hoyko0+96S18Pe0y7oAABIHSURBVOsG0MUQEGG02nuMgEuGgV+Q2ZE2ab9m5jJ6/k+EBvjy8T1DCW4mlS/C80lCr6/cU0aXTNIaSP4G8s+CxRciLy2veW8pfblm2JCaye1LNjKkWzhvTLZK5YvwePVO6EqpkcDLgAVYorWeV2n+5cBLQCwwXmv9YU3rdKuEbq+40Gixl9a8Z6YY01tF2U6sXgsdrOBlMTfOJmT5pl+Z8/FO7hwayRM3SOWL8Gz1SuhKKQuQBFwNpAGbgQla6z12y0QCLYAHgZUendAry0gpT+6H1xtdM83DoPs1tq6Z4eDfwuwoPd7cz/fwxs8H+efNMUwY0MnscIRoMBdL6I6M5TIASNFap9pWthwYDZQldK31Idu8pjfOaXg3CL8PhtwH58/AgW+Nksj9a4zxZrx8IHJoec17aBezI/ZIj4zqxYGT2fz1011EhgUw+JIws0MSotE50uHYHjhi9zrNNq3WlFLTlVKJSqnEkyc98BZjzUIgeiyMXQyzD8Cda2DwPXDuOHw5B/4VD/83AL5+3GjNFxeZHbHH8LZ48cptfekc1pwZ72zhcGaO2SEJ0ega9QyS1nqR1tqqtbZGREQ05qYbn8UbOg+Bq+fCfZtg5jYYOQ9atIX/vgpvXgvPd4OP7oJdHxmte1EvLfx9eH1yfwCmLk3kXF6hyREJ0bgc6XI5CthfXdPBNk3URmhXGDTDeOSdM8aYSfrSGHNm5wrw8oZOg41umZ7XQtglZkfsliLDA3h1YgKTXt/EzPe28frk/li8ZMRN0TQ4clLUG+Ok6HCMRL4ZuE1rvbuKZf8NfNGkTorWV0mxMTpk6YnVE7ZTE2HdyvvdOw0Ci9RY18Y7Gw/z6Ce7mHppF/56fZTZ4QjhNM4oWxyFUZZoAd7QWv9DKTUXSNRar1RK9Qc+AVoCecBvWuuL1o9JQq/G6cPGxUxJX8KhH6G4APyDodtVRklkt+HQPNTsKN3Ckyt38+/1h3hmbAy39pfKF+EZ5MIid5WfBanfG1UzyWsh5yQoi9FiL615D+8uN/GoRlFxCXf+ezMbUjN5e+pABnaVyhfh/iShe4KSEji2tXykyPSdxvTQrnZdM4PB29fcOF3M2fOFjHn1Z87kFvLZvUPpGCo3ORHuTRK6JzqbVp7cD/4Axfng18Lokukx0riwSbpmADiYkcNN83+mdQs/PpoxhCB/OR8h3JckdE9XkGN0zSR9afS/Z6eD8oIOA4xhgHuMhIheTbprZn1KBne8sYkrekSweJJVKl+E25KE3pSUlMDx7eVVM8d/MaaHdLaVRI6EzkPB28/cOE3w1obD/PXTXUy/vCuPjOptdjhC1El9L/0X7sTLC9onGI9hj8C5Y+VVM1uXwqaF4BsIl/yuvGsm0MMv8rK5Y1BnktOzWPRDKt1bBfJ7q9y8RHgWaaE3JQW5Rn97addM1jFAGaNDlp5Ybd3Ho7tmiopLmPLmZjYezOTduwbRP1LOMwj3Il0u4kJaw287bDfP/tKooAEI7lheEhl5Kfj4mxtnAziba6t8OS+VL8L9SEIXNcv6zRiGYP+XkLoOCnPBJ8C4M1OPEcadmoJamx2l06SezOam+T/TNrgZH90zhEA/6X0U7kESuqidwjzjKtXSsshzacb09v1sXTMjoE2s23fN/Jh8kilvbmZYzwgW3iGVL8I9SEIXdac1pO8uv79qWiKgIahd+c2zu1wOPs3MjrROlq4/xBMrd/M/V3TlL9dK5YtwfVLlIupOKWgTbTwunw3ZJyD5ayPB7/wAtrwJ3s2g65VGSWT3EcYQwW5i0uDOJKVnsfA/qXRvFcS4fh3MDkmIOpOELmonsBX0nWg8ivLh0E+2ssg1xgOgbXx5zXubOKOU0kUppXjyxj4czMjhkY930iW8Of06S+WLcE/S5SKcQ2s4uc+49V7Sl3BkE6AhsI2tamak0Yr3dc2KkjO5Bdw0/2ey8or47L6hdGjpmnEKIX3oovHlZBpVM0lfQsq3UJAF3v5Gf3vpidVg1+reSDmRzZhXf6ZtsD9jEzoQFuhHWKAv4QF+hAf5Ehrgi5+3xewwRRMnCV2Yq6gAfl1vq3lfA6cPGdPbxNiS+7XQrq9LdM38kHSSP763jbPnq759XZC/N+GBfoQF+Bo/A30JC/QjPNCXsADbT9vr4GY+KDevBBKuRxK6cB1aQ0ZSeUnkkQ2gSyCgFfS4xtY1Mwz8Ak0MUZNTUExmdj4Z2QUVfmbmFJCRnU9mtu1nTgGncwuo6t/I20sRapf4Sw8EpQnf/oAQFuCLv4+0/kXNJKEL15V7ClK+sd1f9RvIPwsWX4i8zCiJ7DECQlz7bkNFxSWczi0kMyefjKwC42fpAcCW+DNySg8M+eQVllS5niA/7woJPizQj4jS13bfAMID/Qhu5oOX1M03SZLQhXsoLoRfN5SPFJmZYkxv1ae85r19P/By75ZsbkERGVkFZOQYCb+05X8yy/hZeiDIzMnnVE4BJVX8i1psrf/Srp9wu8QfHmD3jcD2U1r/nkMSunBPGSnlyf3wetDF0DzcGCGyp61rxr+F2VE2qOISzencgrLEn5FTQEZWPpk5pa3/0m8ExuvcguIq1xPgaynv67fr86+c+MMCfAlp7itXzbowSejC/Z0/Y+uaWWtUz+SdAS8fYwCx0pr3lpFmR2m63IIiW+u+PPEb3T8VE39GdgGncvKrbP17KQgtO8FbMfFXPhCEB/rRzFda/41JErrwLMVFkLapvOY9I8mYHtGrfBjgjgPcvmumoZWUaM6cLzT6+LMrdv9kVDoRnJldQHZ+UZXrae5ruaCP/8IDgfG8pbT+600SuvBsmQdsI0WugcM/Q0kRNAuF7lcbyb3bcPAPNjtKt3e+oLisq6e05X/hgcCYdiqngOIqmv9KQWhzX7uuntIS0NKuoIrnAZr7WqT0sxJJ6KLpyDsLB76zDUewFs6fAi9v6DykvPUedonZUXq8khLN2fOFVSb+8oqf8hPAWdW0/v19vMoSf3iAr13df8Xun7BAX0Kb++JtMf9ahoYmCV00TSXFxuiQSWuMmveTe43pYd3Lb57dcRBYZEgjs+UVFnOqihr/0sSfYVf5k5ldQFE1rf+WzX1tJZ+lZZ/lJaDGt4LybwYBbtr6l4QuBBhXqCZ9ZST4Qz9BcYHRFdPtaqMksttwaNbS7ChFDbQ2Wv8V+/jzOZldseQzM7uAk9n5ZOVV3fr38/aq9qKv8mnG65YBvvi4SOtfEroQleVnwYF15fdXzc0AZYFOg427NDULMapovLyNh8W7/LmXj3HC1WI33/5hsc0vW9a70vK2+W7YOnRH+UVG67/sIq/qrvq1HQgKi6vOiSHNfS4c9sE2zo/9sA9hgb4E+Xk3WOu/3gldKTUSeBmwAEu01vMqzfcDlgH9gEzgVq31oYutUxK6cBklJXB0S3nNe/quxtmusk/ylnocQKpYtuzAUtuDTeVlHYztYttTXm5z8NJacy6vqOKwDzkVr/rNzC6/KKy6MX98vb1sff4VE3/pyd5+nVvSOSygTjHW6wYXSikLMB+4GkgDNiulVmqt99gtNhU4rbXuppQaDzwD3FqnaIVobF5e0LG/8Rj+V+PEalG+ceVqSdGFj2L714VVTLNftnQdxdUsW2ibV92ydvNL7OYX5NQQm92yZeutOvk0zj529ADSyAebSttTXhaCLT4Ee3nT1ccbwrwhonTZZuAVWGF7BdqL03klZOQWk3Fek5lTWPGbgC3xJ/2WRUZ2AQXFxrAPT4+JqXNCvxhHzgYNAFK01qkASqnlwGjAPqGPBp60Pf8Q+D+llNJm9ecIUR+eXOJYdnCwP1gUVnFAqO5gU/lgVcXBptplixzbXuX3F+U5vqz9oxH4Aq1tD4Oq+gDi642O8KZEWSjGQoH3Q4DzxyhyJKG3B47YvU4DBla3jNa6SCl1FggDMpwRpBDCSbwstguu/MyOpGFpfeHBpjbfjGr8tlPdwa36g5UqKcZSUoSluBDfkIgG+bUbtV5LKTUdmA7QqZNrj6AnhHBjShndLxZvwN/saBqNI3U4R4GOdq872KZVuYxSyhsIxjg5WoHWepHW2qq1tkZENMwRSgghmipHEvpmoLtSqotSyhcYD6ystMxKYLLt+TjgO+k/F0KIxlVjl4utT/w+YC1G2eIbWuvdSqm5QKLWeiXwOvCWUioFOIWR9IUQQjQih/rQtdargdWVpj1u9zwP+L1zQxNCCFEbrnEtqxBCiHqThC6EEB5CEroQQngISehCCOEhTBttUSl1Ejhcx7eH45pXoUpctSNx1Z6rxiZx1U594uqsta7yQh7TEnp9KKUSqxttzEwSV+1IXLXnqrFJXLXTUHFJl4sQQngISehCCOEh3DWhLzI7gGpIXLUjcdWeq8YmcdVOg8Tlln3oQgghLuSuLXQhhBCVSEIXQggP4XIJXSk1Uim1XymVopSaU8V8P6XU+7b5G5VSkXbz/mKbvl8pNaKR4/qTUmqPUmqHUupbpVRnu3nFSqnttkfloYcbOq4pSqmTdtufZjdvslIq2faYXPm9DRzXi3YxJSmlztjNa8j99YZS6oRSqso7QSvDv2xx71BKJdjNa5D95UBME22x7FRKrVdKxdnNO2Sbvl0p5fS7rjsQ25VKqbN2f6/H7eZd9DPQwHHNtotpl+0zFWqb1yD7TCnVUSm1zpYHdiul7q9imYb9fGmtXeaBMTzvAaArxu36fgGiKi1zD/Ca7fl44H3b8yjb8n5AF9t6LI0Y1zCgue35jNK4bK+zTdxfU4D/q+K9oUCq7WdL2/OWjRVXpeX/iDEsc4PuL9u6LwcSgF3VzB8FrAEUMAjY2Aj7q6aYhpRuC7i2NCbb60NAuIn760rgi/p+BpwdV6Vlb8C4R0OD7jOgLZBgex4EJFXx/9igny9Xa6GX3ZBaa10AlN6Q2t5oYKnt+YfAcKWUsk1frrXO11ofBFJs62uUuLTW67TWubaXGzDu7NTQHNlf1RkBfK21PqW1Pg18DYw0Ka4JwHtO2vZFaa1/wBizvzqjgWXasAEIUUq1pQH3V00xaa3X27YJjffZKt12TfurOvX5bDo7rkb5fGmtj2utt9qeZwF7Me63bK9BP1+ultCruiF15R1S4YbUQOkNqR15b0PGZW8qxlG4lL9SKlEptUEpdZOTYqpNXGNtX+8+VEqV3k7QJfaXrWuqC/Cd3eSG2l+OqC72htxftVH5s6WBr5RSW5Rxz14zDFZK/aKUWqOU6mOb5hL7SynVHCMxfmQ3ucH3mTK6gvsCGyvNatDPV6PeJLopUErdDliBK+wmd9ZaH1VKdQW+U0rt1FofaKSQPgfe01rnK6X+B+Pbze8aaduOGA98qLUutptm5v5yWUqpYRgJ/VK7yZfa9lUr4Gul1D5b67WxbMX4e2UrpUYBnwLdG3H7NbkB+Flrbd+ab9B9ppQKxDiAPKC1Pues9TrC1Vro9bkhtSPvbci4UEpdBTwK3Ki1zi+drrU+avuZCnyPceRulLi01pl2sSwB+jn63oaMy854Kn0dbsD95YjqYm/I/VUjpVQsxt9vtNa67AbsdvvqBPAJzutmdIjW+pzWOtv2fDXgo5QKx+T9Zediny+n7zOllA9GMn9Ha/1xFYs07OfL2ScG6nlSwRvjZEAXyk+k9Km0zL1UPCm6wva8DxVPiqbivJOijsTVF+MkUPdK01sCfrbn4UAyTjo55GBcbe2ejwE26PKTMAdt8bW0PQ9trLhsy/XCOEGlGmN/2W0jkupP8l1HxZNWmxp6fzkQUyeMc0JDKk0PAILsnq8HRjpzXzkQW5vSvx9GYvzVtu8c+gw0VFy2+cEY/ewBjbHPbL/3MuCliyzToJ8vp/7hnbRTRmGcHT4APGqbNhej1QvgD3xg+4BvArravfdR2/v2A9c2clzfAOnAdttjpW36EGCn7QO9E5jayHH9E9ht2/46oJfde/9g248pwJ2NGZft9ZPAvErva+j99R5wHCjE6KecCtwN3G2br4D5trh3AtaG3l8OxLQEOG332Uq0Te9q20+/2P7GjzpzXzkY2312n68N2B10qvoMNFZctmWmYBRK2L+vwfYZRleYBnbY/a1GNebnSy79F0IID+FqfehCCCHqSBK6EEJ4CEnoQgjhISShCyGEh5CELoQQHkISuhBCeAhJ6EII4SH+H6p5pGFiW1xBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "def plot_history(history):\n",
    "    plt.plot(history.history['loss'],label=\"Training set loss\")\n",
    "    plt.plot(history.history['val_loss'],label=\"Validation set loss\")\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "plot_history(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Seems to no longer be improving."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prediction\n",
    "\n",
    "For prediction, we'll vectorize the given text normally, and initialize the decoder inputs to a vector with an initial start symbol and zeros otherwise.\n",
    "\n",
    "We'll then predict outputs, take the first new predicted character, place that back into the decoder inputs, and iterate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(text):\n",
    "    encoder_X = vectorize([text], input_tokenizer, INPUT_LENGTH, quiet=True)\n",
    "    decoder_X = np.zeros(shape=(1, OUTPUT_LENGTH))\n",
    "    decoder_X[0,0] = START_INDEX\n",
    "    predictions = []\n",
    "    for i in range(1, OUTPUT_LENGTH):\n",
    "        prediction = model.predict([encoder_X, decoder_X])[0][i].argmax()\n",
    "        predictions.append(prediction)\n",
    "        decoder_X[0,i] = prediction\n",
    "    pred_chars = []\n",
    "    for i in predictions:\n",
    "        if i == 0:\n",
    "            break\n",
    "        pred_chars += output_tokenizer.index_word[i]\n",
    "    return ''.join(pred_chars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test with a few cases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. helmikuuta 2003 → 01.02.2003\n",
      "Toukokuun 7. päivä 1995 → 07.05.1995\n",
      "9. päivä huhtikuuta 2020 → 09.04.2020\n"
     ]
    }
   ],
   "source": [
    "test_inputs = ['1. helmikuuta 2003', 'Toukokuun 7. päivä 1995', '9. päivä huhtikuuta 2020']\n",
    "for text in test_inputs:\n",
    "    print(text, '→', predict(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The coverage of the training data is not comprehensive:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. tammikuuta vuonna 800 → 01.08.2006\n",
      "vuoden 2020 ensimmäinen päivä → 02.07.2006\n"
     ]
    }
   ],
   "source": [
    "test_inputs = ['1. tammikuuta vuonna 800', 'vuoden 2020 ensimmäinen päivä']\n",
    "for text in test_inputs:\n",
    "    print(text, '→', predict(text))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}