seq2seq_katakana.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sequence-to-sequence RNN\n",
    "\n",
    "This notebook shows an example of a character-level sequence-to-sequence recurrent neural network model using Keras.\n",
    "\n",
    "The implementation and example data draw on [A ten-minute introduction to sequence-to-sequence learning in Keras](https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html) and [English to Katakana using a Sequence-to-Sequence model](https://github.com/wanasit/katakana)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration\n",
    "\n",
    "The following variables configure a few aspects of the data, model, and training process. To adapt this example to a different dataset, you'll probably want to change these to match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Maximum number of examples to read\n",
    "MAX_EXAMPLES = 100000\n",
    "\n",
    "# Maximum length of input sequence in characters\n",
    "INPUT_LENGTH = 20\n",
    "\n",
    "# Maximum length of output sequence in characters\n",
    "OUTPUT_LENGTH = 20\n",
    "\n",
    "# Number of epochs to train for\n",
    "EPOCHS = 8\n",
    "\n",
    "# Training batch size\n",
    "BATCH_SIZE = 64\n",
    "\n",
    "# Size of character embeddings\n",
    "EMBEDDING_DIM = 100\n",
    "\n",
    "# Size of RNN states\n",
    "RNN_UNITS = 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download dataset\n",
    "\n",
    "We'll be using the English to Japanese katakana characters dataset from [English to Katakana using a Sequence-to-Sequence model](https://github.com/wanasit/katakana)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File ‘data.csv’ already there; not retrieving.\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!wget -nc https://raw.githubusercontent.com/wanasit/katakana/master/dataset/data.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Paranthropus → パラントロプス\n",
      "Jarret Martin → ジャレット・マーティン\n",
      "Ildefons Lima → イルデフォンス・リマ・ソラ\n",
      "Joseph Calleia → ジョセフ・カレイア\n",
      "Rorschach → ロールシャッハ\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "\n",
    "def load_data(fn, separator=',', has_header_line=True, max_examples=None):\n",
    "    data = []\n",
    "    with open(fn) as f:\n",
    "        if has_header_line:\n",
    "            next(f)    # skip header\n",
    "        for line in f:\n",
    "            if max_examples is not None and len(data) >= max_examples:\n",
    "                break\n",
    "            line = line.rstrip('\\n')\n",
    "            input_text, output_text = line.split(separator)\n",
    "            data.append([input_text, output_text])\n",
    "    return data\n",
    "\n",
    "\n",
    "data = load_data('data.csv', max_examples=MAX_EXAMPLES)\n",
    "\n",
    "random.seed(1234)    # make random.shuffle() repeatable\n",
    "random.shuffle(data)\n",
    "\n",
    "input_texts = [input_text for input_text, output_text in data]\n",
    "output_texts = [output_text for input_text, output_text in data]\n",
    "\n",
    "\n",
    "# Have a look at the source data\n",
    "for i in range(5):\n",
    "    print(input_texts[i], '→', output_texts[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vectorize text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tokenize on the character level with lowercasing. To allow different input and output alphabets, create separate tokenizers for each.\n",
    "\n",
    "We're slightly abusing the tokenizer's out-of-vocabulary item support here to introduce a special `<START>` token into the mapping. As we're not restricting the number of words, this token will never be output by the tokenizer, so we're free to use it for our purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "\n",
    "\n",
    "input_tokenizer = Tokenizer(lower=True, char_level=True)\n",
    "output_tokenizer = Tokenizer(lower=True, char_level=True, oov_token='<START>')\n",
    "\n",
    "input_tokenizer.fit_on_texts(input_texts)\n",
    "output_tokenizer.fit_on_texts(output_texts)\n",
    "\n",
    "# Remember these\n",
    "INPUT_VOCAB_SIZE = max(input_tokenizer.word_index.values()) + 1\n",
    "OUTPUT_VOCAB_SIZE = max(output_tokenizer.word_index.values()) + 1\n",
    "START_INDEX = output_tokenizer.word_index['<START>']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at those mappings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries in input mapping: 134\n",
      "{' ': 7,\n",
      " 'a': 1,\n",
      " 'e': 2,\n",
      " 'i': 5,\n",
      " 'l': 8,\n",
      " 'n': 4,\n",
      " 'o': 6,\n",
      " 'r': 3,\n",
      " 's': 9,\n",
      " 't': 10}\n",
      "Number of entries in output mapping: 88\n",
      "{'<START>': 1,\n",
      " 'ア': 8,\n",
      " 'ス': 6,\n",
      " 'ト': 10,\n",
      " 'ラ': 9,\n",
      " 'リ': 7,\n",
      " 'ル': 5,\n",
      " 'ン': 4,\n",
      " '・': 3,\n",
      " 'ー': 2}\n"
     ]
    }
   ],
   "source": [
    "from pprint import pprint    # pretty-printer\n",
    "\n",
    "\n",
    "def truncate_dict(d, count=10):\n",
    "    # Returns at most count items from the given dictionary.  \n",
    "    return dict(i for i, _ in zip(d.items(), range(count)))\n",
    "\n",
    "\n",
    "print('Number of entries in input mapping:', len(input_tokenizer.word_index))\n",
    "pprint(truncate_dict(input_tokenizer.word_index))\n",
    "print('Number of entries in output mapping:', len(output_tokenizer.word_index))\n",
    "pprint(truncate_dict(output_tokenizer.word_index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Map input and output texts to integer sequences using the tokenizers, then pad and truncate the sequences to desired input and output lengths."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text: Paranthropus\n",
      "Sequence: [19, 1, 3, 1, 4, 10, 13, 3, 6, 19, 15, 9]\n",
      "Padded: [19  1  3  1  4 10 13  3  6 19 15  9  0  0  0  0  0  0  0  0]\n",
      "Mapped back: ['p', 'a', 'r', 'a', 'n', 't', 'h', 'r', 'o', 'p', 'u', 's', '-', '-', '-', '-', '-', '-', '-', '-']\n",
      "Text: パラントロプス\n",
      "Sequence: [52, 9, 4, 10, 18, 48, 6]\n",
      "Padded: [52  9  4 10 18 48  6  0  0  0  0  0  0  0  0  0  0  0  0  0]\n",
      "Mapped back: ['パ', 'ラ', 'ン', 'ト', 'ロ', 'プ', 'ス', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-']\n",
      "type(encoder_X): <class 'numpy.ndarray'>\n",
      "encoder_X.shape: (100000, 20)\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "\n",
    "def vectorize(texts, tokenizer, maxlen, quiet=False):\n",
    "    # This bit does the work\n",
    "    sequences = tokenizer.texts_to_sequences(texts)\n",
    "    padded = pad_sequences(sequences, maxlen=maxlen, padding='post')\n",
    "    \n",
    "    # This just prints out the first input and its vectorized versions\n",
    "    if not quiet:\n",
    "        print('Text:', texts[0])\n",
    "        print('Sequence:', sequences[0])\n",
    "        print('Padded:', padded[0])\n",
    "        print('Mapped back:', [tokenizer.index_word.get(i, '-') for i in padded[0]])\n",
    "    \n",
    "    return padded\n",
    "\n",
    "\n",
    "encoder_X = vectorize(input_texts, input_tokenizer, INPUT_LENGTH)\n",
    "decoder_Y = vectorize(output_texts, output_tokenizer, OUTPUT_LENGTH)\n",
    "\n",
    "# This creates numpy arrays:\n",
    "print('type(encoder_X):', type(encoder_X))\n",
    "print('encoder_X.shape:', encoder_X.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In prediction, the decoder will receive its last output as input at each timestep. During training, we'll use _teacher forcing_, where the decoder is instead given the correct previous output. To implement this, the output sequence (`decoder_Y`) is shifted one character forward, and the special start symbol is placed first in the sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "decoder_X: [ 1 52  9  4 10 18 48  6  0  0  0  0  0  0  0  0  0  0  0  0]\n",
      "Mapped back: ['<START>', 'パ', 'ラ', 'ン', 'ト', 'ロ', 'プ', 'ス', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-']\n",
      "decoder_Y: [52  9  4 10 18 48  6  0  0  0  0  0  0  0  0  0  0  0  0  0]\n",
      "Mapped back: ['パ', 'ラ', 'ン', 'ト', 'ロ', 'プ', 'ス', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-']\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "\n",
    "decoder_X = np.zeros_like(decoder_Y)\n",
    "decoder_X[:,1:] = decoder_Y[:,:-1]\n",
    "decoder_X[:,0] = START_INDEX\n",
    "\n",
    "print('decoder_X:', decoder_X[0])\n",
    "print('Mapped back:', [output_tokenizer.index_word.get(i, '-') for i in decoder_X[0]])\n",
    "print('decoder_Y:', decoder_Y[0])\n",
    "print('Mapped back:', [output_tokenizer.index_word.get(i, '-') for i in decoder_Y[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert output integer values standing for vocabulary items (characters) into a one-hot representation for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "decoder_Y[0][0]: 52\n",
      "one_hot_decoder_Y[0][0]:\n",
      "[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "one_hot_decoder_Y[0][0].argmax(): 52\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.utils import to_categorical\n",
    "\n",
    "\n",
    "one_hot_decoder_Y = to_categorical(decoder_Y)\n",
    "\n",
    "print('decoder_Y[0][0]:', decoder_Y[0][0])\n",
    "print('one_hot_decoder_Y[0][0]:')\n",
    "print(one_hot_decoder_Y[0][0])\n",
    "print('one_hot_decoder_Y[0][0].argmax():', one_hot_decoder_Y[0][0].argmax())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build model\n",
    "\n",
    "Note the following:\n",
    "\n",
    "* `mask_zero=True` for the embedding makes the model ignore padding (see [Masking and padding with Keras](https://www.tensorflow.org/guide/keras/masking_and_padding))\n",
    "* `return_state=True` for the encoder RNN returns the state of the last timestep in addition to output (see https://keras.io/layers/recurrent/).\n",
    "* As we're using an LSTM, we get two separate state values, _h_ and _c_ (see below)\n",
    "* `return_sequences=True` for the decoder RNN returns the output from each time step, not only the last\n",
    "\n",
    "<img src=\"https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-chain.png\" style=\"width: 50%\">\n",
    "\n",
    "(LSTM illustration from https://colah.github.io/posts/2015-08-Understanding-LSTMs/ )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/smp/Library/Python/3.7/lib/python/site-packages/tensorflow_core/python/keras/initializers.py:119: calling RandomUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
      "WARNING:tensorflow:From /Users/smp/Library/Python/3.7/lib/python/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n",
      "WARNING:tensorflow:From /Users/smp/Library/Python/3.7/lib/python/site-packages/tensorflow_core/python/keras/backend.py:3872: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            [(None, 20)]         0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_2 (InputLayer)            [(None, 20)]         0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding (Embedding)           (None, 20, 100)      13500       input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 20, 100)      8900        input_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "lstm (LSTM)                     [(None, 100), (None, 80400       embedding[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "lstm_1 (LSTM)                   (None, 20, 100)      80400       embedding_1[0][0]                \n",
      "                                                                 lstm[0][1]                       \n",
      "                                                                 lstm[0][2]                       \n",
      "__________________________________________________________________________________________________\n",
      "time_distributed (TimeDistribut (None, 20, 89)       8989        lstm_1[0][0]                     \n",
      "==================================================================================================\n",
      "Total params: 192,189\n",
      "Trainable params: 192,189\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.layers import Input, Embedding, LSTM, TimeDistributed, Dense\n",
    "from tensorflow.keras.models import Model, load_model\n",
    "\n",
    "\n",
    "def build_seq2seq_model(input_length, output_length,\n",
    "                        input_vocab_size, output_vocab_size,\n",
    "                        embedding_dim=EMBEDDING_DIM, rnn_units=RNN_UNITS):\n",
    "    encoder_input = Input(shape=(input_length,))\n",
    "    encoder_embedding = Embedding(input_vocab_size, embedding_dim, mask_zero=True)(encoder_input)\n",
    "    encoder_output, encoder_h, encoder_c = LSTM(\n",
    "        rnn_units,\n",
    "        return_sequences=False,\n",
    "        return_state=True,\n",
    "        unroll=True\n",
    "    )(encoder_embedding)\n",
    "    encoder_states = [encoder_h, encoder_c]\n",
    "\n",
    "    decoder_input = Input(shape=(output_length,))\n",
    "    decoder_embedding = Embedding(output_vocab_size, embedding_dim, mask_zero=True)(decoder_input)\n",
    "    decoder_rnn = LSTM(rnn_units, return_sequences=True, unroll=True)(\n",
    "        decoder_embedding,\n",
    "        initial_state=encoder_states\n",
    "    )\n",
    "    decoder_output = TimeDistributed(Dense(output_vocab_size, activation=\"softmax\"))(decoder_rnn)\n",
    "\n",
    "    model = Model(inputs=[encoder_input, decoder_input], outputs=[decoder_output])\n",
    "    return model\n",
    "\n",
    "\n",
    "model = build_seq2seq_model(\n",
    "    input_length=INPUT_LENGTH,\n",
    "    output_length=OUTPUT_LENGTH,\n",
    "    input_vocab_size=INPUT_VOCAB_SIZE,\n",
    "    output_vocab_size=OUTPUT_VOCAB_SIZE\n",
    ")\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "As we have one-hot encoded our outputs, we'll use `categorical_crossentropy` loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(\n",
    "    optimizer='adam',\n",
    "    loss='categorical_crossentropy'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 80000 samples, validate on 20000 samples\n",
      "Epoch 1/8\n",
      "80000/80000 [==============================] - 110s 1ms/sample - loss: 1.3737 - val_loss: 1.1209\n",
      "Epoch 2/8\n",
      "80000/80000 [==============================] - 113s 1ms/sample - loss: 0.9445 - val_loss: 0.8099\n",
      "Epoch 3/8\n",
      "80000/80000 [==============================] - 114s 1ms/sample - loss: 0.7404 - val_loss: 0.6793\n",
      "Epoch 4/8\n",
      "80000/80000 [==============================] - 110s 1ms/sample - loss: 0.6404 - val_loss: 0.6092\n",
      "Epoch 5/8\n",
      "80000/80000 [==============================] - 104s 1ms/sample - loss: 0.5815 - val_loss: 0.5678\n",
      "Epoch 6/8\n",
      "80000/80000 [==============================] - 103s 1ms/sample - loss: 0.5415 - val_loss: 0.5340\n",
      "Epoch 7/8\n",
      "80000/80000 [==============================] - 103s 1ms/sample - loss: 0.5119 - val_loss: 0.5130\n",
      "Epoch 8/8\n",
      "80000/80000 [==============================] - 96s 1ms/sample - loss: 0.4885 - val_loss: 0.4906\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(\n",
    "    x=[encoder_X, decoder_X], \n",
    "    y=[one_hot_decoder_Y],\n",
    "    batch_size=BATCH_SIZE,\n",
    "    epochs=EPOCHS,\n",
    "    validation_split=0.2\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the training history:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3dd3hVVdr38e9KD+khBVJIkZaQhBBCE+lFVGawMDaUAQTGMtYZlfFxRscpr+M4iOODKCD6qCgi1hEsIDiASAkgNZBQAiSBNEjvyXr/2CGCkBCSfTg5J/fnunJBcs5Z607QH4t11r630lojhBDC9jlYuwAhhBDmkEAXQgg7IYEuhBB2QgJdCCHshAS6EELYCSdrTRwQEKAjIyOtNb0QQtik7du352utAy/22CUDXSm1BJgI5Gqt45p53gDgB+B2rfWKS40bGRlJSkrKpZ4mhBDiHEqpY0091pItl7eACZeYwBH4B/DNZVUmhBDCNJcMdK31euD0JZ72IPARkGtGUUIIIS5fm98UVUqFAjcBC1rw3NlKqRSlVEpeXl5bpxZCCHEOM94UnQc8qbWuV0o1+0St9UJgIUBycrL0HBCiBWpqasjMzKSystLapYgryM3NjbCwMJydnVv8GjMCPRlY1hDmAcD1SqlarfWnJowtRIeXmZmJl5cXkZGRXGrRJOyD1pqCggIyMzOJiopq8evaHOha68bZlFJvAV9ImAthnsrKSgnzDkYpRefOnbncremWHFt8HxgJBCilMoFnAGcArfVrl1+qEOJySZh3PK35M79koGut72jpYFrraZddwWU6klfKO5uP8YfrYnBxkgtdhRDiLJtLxIyCMt78PoOVe7KtXYoQHUJBQQGJiYkkJibSpUsXQkNDGz+vrq5u0RjTp0/n4MGDzT5n/vz5LF261IySL8vatWvZvHnzRR9bvHgxjzzyyBWuqPWsdul/a43sGUSPIE8Wrj/KjYmh8k9RISysc+fO/PjjjwA8++yzeHp68vvf//6852it0Vrj4HDxNeKbb755yXkeeOCBthfbCmvXriUgIIDBgwdbZX4z2dwK3cFBMWtYNKkni/n+UIG1yxGiwzp06BCxsbFMmTKFPn36cPLkSWbPnk1ycjJ9+vThueeea3zuNddcw48//khtbS2+vr7MmTOHvn37MmTIEHJzjesRn376aebNm9f4/Dlz5jBw4EB69erFpk2bACgrK+OWW24hNjaWyZMnk5yc3PiXzbkef/xxYmNjSUhI4MknnwQgJyeHm2++meTkZAYOHMjmzZs5fPgwixcv5p///CeJiYmN81zM0aNHGTVqFAkJCYwbN47MzEwAli1bRlxcHH379mXUqFEA7NmzhwEDBpCYmEhCQgJHjhwx4Sd+aTa3QgeY1C+EF74+yMINR7imR4C1yxHiivnzf/axP7vY1DFjQ7x55hd9WvXaAwcO8Pbbb5OcnAzA888/j7+/P7W1tYwaNYrJkycTGxt73muKiooYMWIEzz//PI899hhLlixhzpw5F4yttWbr1q18/vnnPPfcc3z11Ve88sordOnShY8++ohdu3aRlJR0wetycnJYtWoV+/btQylFYWEhAA899BBPPPEEgwcPJiMjg4kTJ7J3715mzpxJQEDAJbdW7r//fmbOnMmUKVNYuHAhjzzyCCtWrODPf/4z3333HcHBwY1zvfrqq/z+97/ntttuo6qqiit1q0+bW6EDuDo5Mn1oJOvT8kg9ae5/3EKIlrvqqqsawxzg/fffJykpiaSkJFJTU9m/f/8Fr3F3d+e6664DoH///mRkZFx07JtvvvmC52zcuJHbb78dgL59+9Knz4V/Efn7++Pg4MCsWbP45JNP8PDwAGDNmjXce++9JCYmcuONN3LmzBkqKipa/L1u2bKlce6pU6eyYcMGAIYOHcrUqVNZvHgx9fX1AFx99dX89a9/5YUXXuDEiRO4ubm1eJ62sMkVOsCUQd3437WHWLzhKP+6ta+1yxHiimjtStpSzoYlQHp6Oi+//DJbt27F19eXu+6666JXt7q4uDT+3tHRkdra2ouO7erqesnnXIyzszMpKSmsXr2aDz/8kAULFvDNN980rvjPnd8MixYtYsuWLXzxxRckJSWxc+dO7r77boYMGcLKlSuZMGECS5YsYfjw4abOezE2uUIH8O3kwm0Dwvl8VxaniuSSaCGsrbi4GC8vL7y9vTl58iRff/216XMMHTqU5cuXA8Y+9cX+BVBSUkJxcTETJ07kpZdeYufOnQCMHTuW+fPnNz7v7N67l5cXJSUll5x78ODBjXO/++67jQF95MgRBg8ezF/+8hf8/PzIysriyJEjdO/enYcffpiJEyeye/futn3jLWSzgQ4wY2gUdfWaNzcdtXYpQnR4SUlJxMbG0rt3b6ZOncrQoUNNn+PBBx8kKyuL2NhY/vznPxMbG4uPj895zykqKuKGG26gb9++jBgxgrlz5wLGscjvv/+ehIQEYmNjWbRoEQCTJk1i+fLl9OvXr9k3RefPn8/ChQtJSEjggw8+4KWXXgLg0UcfJT4+nvj4eEaNGkVcXBzvvfceffr0ITExkbS0NO666y7TfxYXo67UZv3PJScnazNucPHA0h2sT8tj0x9G4+XW8iY2QtiK1NRUYmJirF1Gu1BbW0ttbS1ubm6kp6czfvx40tPTcXKy2d3jZl3sz14ptV1rnXyx59v8T2H28GhW7jnJB9tOMHNYtLXLEUJYUGlpKWPGjKG2thatNa+//rrdhnlr2PxPom+4LwOj/Hnz+wx+fXUkzo42vYskhGiGr68v27dvt3YZ7ZZdpN/sYdFkFVawas9Ja5cihBBWYxeBPrp3ENGBHizacOSKHeAXQoj2xi4C/Ww7gL1ZxfxwRNoBCCE6JrsIdICb+oUS4OnCovVXpmeCEEK0N3YT6G7OjkwdEsm6g3mk5Vz6IgEhRMuMGjXqgouE5s2bx3333dfs6zw9PQHIzs5m8uTJF33OyJEjudTx5Xnz5lFeXt74+fXXX9/YM+VKycjI4L333mvysbi4uCtaT1PsJtAB7h4cgZuzg6zShTDRHXfcwbJly8772rJly7jjjpbd+yYkJIQVK1a0ev6fB/qqVavw9fVt9Xit0Vygtyd2Feh+Hi7cmhzOpz9mkVss7QCEMMPkyZNZuXJl480sMjIyyM7OZtiwYY3nwpOSkoiPj+ezzz674PXnrmArKiq4/fbbiYmJ4aabbjqvOdZ9993X2Hr3mWeeAeDf//432dnZjBo1qrE1bWRkJPn5+QDMnTuXuLg44uLiGlvvZmRkEBMTw6xZs+jTpw/jx4+/aBOuDz/8sLHt7dnL+Ovq6nj88ccZMGAACQkJvP766wDMmTOHDRs2kJiY2HiF6MVUVlYyffp04uPj6devH+vWrQNg3759DBw4sLGdbnp6OmVlZY1XtMbFxfHBBx9cxp/Kxdn8OfSfu+eaKN7ZfIy3NmXwxITe1i5HCHN9OQdO7TF3zC7xcN3zTT7s7+/PwIED+fLLL5k0aRLLli3j1ltvRSmFm5sbn3zyCd7e3uTn5zN48GB++ctfNnnjmQULFtCpUydSU1PZvXv3ee1v//a3v+Hv709dXR1jxoxh9+7dPPTQQ8ydO5d169YREHB+q+zt27fz5ptvsmXLFrTWDBo0iBEjRuDn50d6ejrvv/8+ixYt4tZbb+Wjjz664PL75557jq+//prQ0NDGLZw33ngDHx8ftm3bRlVVFUOHDmX8+PE8//zzvPjii3zxxRfN/ijnz5+PUoo9e/Zw4MABxo8fT1paGq+99hoPP/wwU6ZMobq6mrq6OlatWkVISAgrV64EjJYFbWVXK3SAiM4eTOjThXc3H6OsquUd2oQQTTt32+Xc7RatNU899RQJCQmMHTuWrKwscnJymhxn/fr1jcGakJBAQkJC42PLly8nKSmJfv36sW/fvos23jrXxo0buemmm/Dw8MDT05Obb765saVtVFQUiYmJQNMteocOHcq0adNYtGgRdXV1AHzzzTe8/fbbJCYmMmjQIAoKCkhPT2/hT8mo6ez317t3byIiIkhLS2PIkCH8/e9/5x//+AfHjh3D3d2d+Ph4Vq9ezZNPPsmGDRsu6EnTGna3QgeYNTyaL/eeYnnKCaYPjbJ2OUKYp5mVtCVNmjSJRx99lB07dlBeXk7//v0BWLp0KXl5eWzfvh1nZ2ciIyMv2jL3Uo4ePcqLL77Itm3b8PPzY9q0aa0a56yzrXfBaL97sS2X1157jS1btrBy5Ur69+/P9u3b0VrzyiuvcO2115733O+++67VtQDceeedDBo0iJUrV3L99dfz+uuvM3r0aHbs2MGqVat4+umnGTNmDH/605/aNI/drdABkrr5kRzhxxsbj1JbV2/tcoSweZ6enowaNYoZM2ac92ZoUVERQUFBODs7s27dOo4dO9bsOMOHD298c3Hv3r2NbWWLi4vx8PDAx8eHnJwcvvzyy8bXNNXedtiwYXz66aeUl5dTVlbGJ598wrBhw1r8PR0+fJhBgwbx3HPPERgYyIkTJ7j22mtZsGABNTU1AKSlpVFWVtbiFrvDhg1rvNF1Wloax48fp1evXhw5coTo6GgeeughJk2axO7du8nOzqZTp07cddddPP744+zYsaPFtTfFLlfoYKzSf/POdr7ad4qJCSHWLkcIm3fHHXdw0003nXfiZcqUKfziF78gPj6e5ORkevdu/n2r++67j+nTpxMTE0NMTEzjSr9v377069eP3r17Ex4efl7r3dmzZzNhwgRCQkIa32QEo13vtGnTGDhwIAAzZ86kX79+Td4B6ecef/xx0tPT0VozZswY+vbtS0JCAhkZGSQlJaG1JjAwkE8//ZSEhAQcHR3p27cv06ZN49FHH73omPfffz/33Xcf8fHxODk58dZbb+Hq6sry5ct55513cHZ2pkuXLjz11FNs27aNxx9/HAcHB5ydnVmwYEGL6m6OzbfPbUpdvWbs3P/i5ebEZw8MbfJNGiHaO2mf23Fdbvtcu9xyAXB0UMwcFsXuzCK2HD1t7XKEEMLi7DbQAW5JCsPfQ9oBCCE6BrsOdKMdQATfHsjlUK60AxC2S7qIdjyt+TO360AHox2Aq5MDizfIfUeFbXJzc6OgoEBCvQPRWlNQUICbm9tlvc5uT7mc1dnTlcn9w/gwJZPHxvckyOvyfkBCWFtYWBiZmZnk5eVZuxRxBbm5uREWFnZZr7H7QAejHcB7W4/zzg/H+N34XtYuR4jL4uzsTFSUXCAnLs3ut1wAogM9GRcTzDubj1FeLe0AhBD2qUMEOsDs4dEUltfwYUqmtUsRQgiL6DCBnhzpT1I3XxZvPEJdvby5JISwPx0m0MFYpZ84XcHX+05ZuxQhhDBdhwr0cbFdiOjcidfXH5EjYEIIu9OhAt3RQTHzmih2nSgk5dgZa5cjhBCmumSgK6WWKKVylVJ7m3h8ilJqt1Jqj1Jqk1Kqr/llmmdy/3D8OjmzUNoBCCHsTEtW6G8BE5p5/CgwQmsdD/wFWGhCXRbj7uLI3YMjWJOaw+G8UmuXI4QQprlkoGut1wNNtivUWm/SWp/dv9gMXN6lTVZw95BInB0deGOjtAMQQtgPs/fQ7wG+bOpBpdRspVSKUirFmpcxB3q5cktSKCu2Z5JfWmW1OoQQwkymBbpSahRGoD/Z1HO01gu11sla6+TAwECzpm6VmcOiqa6t5+0fmr9llhBC2ApTAl0plQAsBiZprQvMGNPSrgr0ZGxMMO/8kEFFdZ21yxFCiDZrc6ArpboBHwN3a63T2l7SlTN7eDRnymtYsUPaAQghbF9Lji2+D/wA9FJKZSql7lFK3auUurfhKX8COgOvKqV+VEpZ7kahJhsQ6UffcF/e2CDtAIQQtu+S7XO11ndc4vGZwEzTKrqClFLMHhbNA+/tYPX+HCbEdbF2SUII0Wod6krRi7m2TzDh/u4s2iAXGgkhbFuHD3QnRwfuGRrF9mNn2H6syeP2QgjR7nX4QAf4VXI4Pu7SDkAIYdsk0AEPVyfuHhzBN/tzOJpfZu1yhBCiVSTQG0y9OgJnBwfe2CirdCGEbZJAbxDk5cZN/UL5MCWTAmkHIISwQRLo55g5LIqq2nre3Xzc2qUIIcRlk0A/R49gL0b3DuLtHzKorJF2AEII2yKB/jOzhkVTUFbNxzuyrF2KEEJcFgn0nxkc7U98qA+LNxyhXtoBCCFsiG0GeqHl9riVUsweHs2R/DLWpOZYbB4hhDCb7QX6rmXw735wcpfFprgurguhvtIOQAhhW2wv0HteC+5+8J+Hod4yb1w6OTpwzzVRbMs4w47jZy79AiGEaAdsL9Dd/WDC85C9E7Ytttg0tw4Ix9vNicWyShdC2AjbC3SAuFvgqtHw7V+gyDKnUTxdnZgyOIKv9p7iWIG0AxBCtH+2GehKwQ1zob4GvnzCYtNMuzoSRwfFko1HLTaHEEKYxTYDHcA/CkY8CQe+gAMrLTJFsLcbkxJDWZ6SyZmyaovMIYQQZrHdQAe4+kEIioVVj0NViUWmmDUsmoqaOpZuOWaR8YUQwiy2HeiOzvCLl6E4C9b93SJT9Orixchegby16Zi0AxBCtGu2HegA4QMheQZsec04+WIBs4dFk19axac7pR2AEKL9sv1ABxjzDHQKMM6m19WaPvyQqzrTJ8SbRdIOQAjRjtlHoLv7wnXPG1ePbl1o+vBn2wEczitj3cFc08cXQggz2EegA/S5GbqPg7V/haJM04e/Pr4rIT5uct9RIUS7ZT+BrhTc8CLoelhl/tl0Z0cHZlwTxZajp9l1otD08YUQoq3sJ9AB/CJh5Bw4uBJSvzB9+NsGhOPl6iRNu4QQ7ZJ9BTrAkAcgOM44m15ZbOrQXm7O3DmoG6v2nOTE6XJTxxZCiLayv0B3dIaJ86DkJKz7m+nDTx8ahYNSvCHtAIQQ7Yz9BTpA+AAYcA9seR2ytps6dBcfN36ZGMLylBMUlks7ACFE+2GfgQ4w5k/gGWyRs+mzhkVTXl3H0i2Wu3OSEEJcLvsNdDcfuO4fcGqPcRWpiWK6ejOsRwBvbcqgqlbaAQgh2gf7DXSA2EnQ41pjL93k+5DOHh5NXkkVn/2Ybeq4QgjRWvYd6GfPpoNx6kWbd9n+Nd0D6N3Fi0Xrj6BNHFcIIVrLvgMdwLcbjHoK0r6C1M9NG/ZsO4D03FK+S8szbVwhhGgt+w90gEH3QXC8cQVpZZFpw05MCKGLtxuLpB2AEKId6BiB7uhk9E0vzTHuQ2oSFycHZlwTyabDBezNMu8vCiGEaI2OEegAYf1h4GzYthgyU0wb9vaB3fB0dZKmXUIIq7tkoCulliilcpVSe5t4XCml/q2UOqSU2q2USjK/TJOMfhq8ujScTa8xZUhvN2fuGBjOyj0nyTwj7QCEENbTkhX6W8CEZh6/DujR8DEbWND2sizEzRuuewFy9sJm88qcPjQKBbz5fYZpYwohxOW6ZKBrrdcDp5t5yiTgbW3YDPgqpbqaVaDpYn4Bva6H7/4fnDHnxs8hvu5MTOjKsq3HKaowZ+UvhBCXy4w99FDgxDmfZzZ87QJKqdlKqRSlVEpenpWO+illrNJRsOr3pp1NnzksmrLqOt7fKu0AhBDWcUXfFNVaL9RaJ2utkwMDA6/k1OfzDYfR/wPp38D+T00ZMi7Uh6HdO/Pm90eprq03ZUwhhLgcZgR6FhB+zudhDV9r3wb+Brr2hS+fhApz7kA0a1g0OcVV/GeXtAMQQlx5ZgT658DUhtMug4EirfVJE8a1rLNn08vy4NvnTBlyRM9AegV7sWiDtAMQQlx5LTm2+D7wA9BLKZWplLpHKXWvUurehqesAo4Ah4BFwP0Wq9ZsIf2MlXrKEjixtc3DKaWYNTyaA6dKWJ+eb0KBQgjRcspaK8nk5GSdkmLeBT6tVlUC8wcZ7XZ/s96441EbVNfWM+yFtfQI8uLdmYNMKlIIIQxKqe1a6+SLPdZxrhRtiqsXXP9PyN0PP/xvm4dzcXJg2tVRbDyUz75saQcghLhyJNABet8AvSfCd/+A022/V+idg7rh4eLI4g1y31EhxJUjgX7WdS+AgyOs/F2bz6b7uDtz24Bu/GdXNtmFFSYVKIQQzZNAP8snFEb/EQ5/C3s/avNwM66JRANvbcpo81hCCNESEujnGjgLuibCV3+AijNtGirMrxM3xHflvS3HKa6UdgBCCMuTQD+Xg6NxNr08H9b8uc3DzRoWTWlVLW/LKl0IcQVIoP9cSCIMvh+2vwnHN7dpqPgwH8bHBjN3dZpcPSqEsDgJ9IsZ+QfwDoP/PAK11W0a6uXb+5Ec4c+jH/zImv05JhUohBAXkkC/GFdPuOFFyEuFH15p01DuLo68MS2Z2BBv7n9vBxvlClIhhIVIoDel13UQ80v47wtwum23l/Nyc+btGQOJDvBg1tspbMtorr28EEK0jgR6c677Bzg4wxePtflsum8nF965ZxBdfdyY8eY2dmea0+FRCCHOkkBvjncIjPkTHFkHe1a0ebhAL1eWzhqETydnpi7ZysFTJSYUKYQQBgn0SxlwD4T2h6/mQHnbt0q6+rizdOYgXJ0cmLJ4C0fzy0woUgghJNAvzcERJs4zLjRa84wpQ0Z09mDpzEForZmyaDOZZ8pNGVcI0bFJoLdE1wQYcj/seBuObTJlyO5BXrx9z0BKq2qZsngLOcWVpowrhOi4JNBbauQfwKdbw9n0KlOG7BPiw1szBpJfUsVdi7dQUGrOuEKIjkkCvaVcPOCGf0H+Qfj+36YNm9TNj8W/HsDx0+VMXbKVogrp+yKEaB0J9MvRczzE3gjr/wkFh00bdshVnXn97v6k5ZQw/c2tlFXVmja2EKLjkEC/XBOeBydX+OLRNp9NP9fIXkG8ckcSuzKLmPl/KVTW1Jk2thCiY5BAv1zeXWHsM3D0v7D7A1OHnhDXhRd/lcDmowXc9+52qmvrTR1fCGHfJNBbo/8MCE2Gr58y5Wz6uW7qF8bfboxn3cE8HvlgJ7V1EupCiJaRQG8NBwejb3plEaz+o+nD3zmoG0/fEMOqPad4YsVu6uvN29oRQtgvCfTW6hIHQ34LO9+FjI2mDz9zWDSPjevJxzuz+ONne9Em7tcLIeyTBHpbjHgSfM09m36uB0d3594RV7F0y3H+vipVQl0I0SwJ9LZw6QQ3vAQF6bBxnunDK6V4ckIvpg6JYNGGo7z8bbrpcwgh7IcEelv1GAtxt8CGFyHf/MBVSvHsL/owuX8Y89aks3C9eeffhRD2RQLdDNf+P3ByN/1s+lkODop/3JLADQld+fuqA7zzQ4bpcwghbJ8Euhm8gmHcs5CxAXa9b5EpHB0U825LZGxMEH/8bB8rtmdaZB4hhO2SQDdL0jQIHwRf/w+UFVhkCmdHB/73ziSGdu/MEyt2sWrPSYvMI4SwTRLoZnFwMPqmVxXDN09bbBo3Z0cWTU0mqZsfD72/k7UHciw2lxDCtkigmyk4Fq5+CHa9B0fXW2yaTi5OLJk+gJiu3tz77g42Hcq32FxCCNshgW62EU+AX6RxNr3Gcjet8HZz5u0ZA4nq7MHMt1PYfszcFgRCCNsjgW42Z3e4YS6cPgwb51p0Kj8PF96ZOZAgL1emLdnG3qwii84nhGjfJNAtofsYiP8VbJgLeWkWnSrIy42lswbj7e7M3W9sIS2nxKLzCSHaLwl0S7n278aVpF88YpGz6ecK9XVn6cxBODs6MGXxFjLyyyw6nxCifWpRoCulJiilDiqlDiml5lzk8W5KqXVKqZ1Kqd1KqevNL9XGeAbBuL/Ase9h0ysWD/XIAA+WzhxEbV09UxZvIauwwqLzCSHan0sGulLKEZgPXAfEAncopWJ/9rSngeVa637A7cCrZhdqk/rdDT3GGy12l90JJZY9Ytgj2It37hlEcWUNUxZtJrfYcm/KCiHan5as0AcCh7TWR7TW1cAyYNLPnqMB74bf+wDZ5pVowxwc4I5lMP5vcHgtvDoI9qyw6Go9LtSHt6YPILekirve2MLpsmqLzSWEaF9aEuihwIlzPs9s+Nq5ngXuUkplAquABy82kFJqtlIqRSmVkpeX14pybZCDI1z9W7h3I3TuDh/dAx/cBaW5Fpuyf4Q/i6cmk1FQztQlWyiurLHYXEKI9sOsN0XvAN7SWocB1wPvKKUuGFtrvVBrnay1Tg4MDDRpahsR0ANmfA3jnoP01TB/EOz92GLTXd09gNfv6s/BUyVMf3Mb5dW1FptLCNE+tCTQs4Dwcz4Pa/jaue4BlgNorX8A3IAAMwq0Kw6OMPRh+M164+KjFdNh+a+hzDJXeo7qHcTLt/dj5/EzzHo7hcqaOovMI4RoH1oS6NuAHkqpKKWUC8abnp//7DnHgTEASqkYjEDvIHsqrRDUG+5ZDWOegYOrjNX6/s8sMtX18V355+S+fH+ogPuX7qC6Vm46LYS9umSga61rgd8CXwOpGKdZ9imlnlNK/bLhab8DZimldgHvA9O03C+teY5OMOwxmP1f8AmD5VNhxQwoN/8S/lv6h/GXG+NYeyCXRz/4kTq56bQQdklZK3eTk5N1SkqKVeZud+pqjFvY/fcf4O4Hv5gHvW8wfZpF64/wt1WpTO4fxgu3JODgoEyfQwhhWUqp7Vrr5Is9JleKtgeOzjDicZj9nXGzjGV3wkezTF+tzxoezSNje7BieybPfL5PbjothJ2RQG9PusTBrHUw8g+w72N4dTAc/NLUKR4e04PZw6N5Z/Mxnv/ygIS6EHZEAr29cXSGkXNg1lroFADv3w6f3AcVhaYMr5TiD9f1Zsqgbry+/givrD1kyrhCCOuTQG+vuvY1tmCGPwG7PzBW62nfmDK0Uoq/TIrj5qRQ5q5OY/GGI6aMK4SwLgn09szJBUb/D8z6Ftx84b1fwWcPQGXb+547OCheuCWB6+O78NeVqSzdcsyEgoUQ1iSBbgtC+sFv/gvXPAY/vgevDoFD37Z5WCdHB+bd1o9RvQJ5+tO9fLwj04RihRDWIoFuK5xcYewzcM8acPGEd2+Gzx+CyuI2Devi5MCCu/ozOKozv/9wF1/uOWlSwUKIK00C3daE9TdaBwx9GHa+AwuuhsPr2jSkmw6Bnu4AABQaSURBVLMji3+dTGK4L799fydzPtpN5plykwoWQlwpEui2yNnNaPI142tj5f7OjfDFY1BV2uohPVydeGvGQO4eHMHHO7IY9eJ3/OmzveRIT3UhbIZcKWrraipg7V/hh/ngGw6T5kPU8DYNmV1YwStrD/FhygkcHRR3D47g3pFXEeDpalLRQojWau5KUQl0e3F8M3x6H5w+AgNnw9hnwcWjbUMWlPPyt+l8sjMTN2dHfn11JL8ZHo1vJxdTShZCXD4J9I6iuhzW/gU2LwC/CJj0KkQObfOwh/NKmbcmnS92Z+Pp4sSMa6K4Z1gU3m7OJhQthLgcEugdTcb38Nn9cOYYDLoXxvwJXDq1ediDp0p4aXUaX+07hY+7M7OHRzPt6kg8XJ1MKFoI0RIS6B1RdRmseRa2LgT/aLhxAXQbbMrQe7OKmLs6jbUHcuns4cJ9I6/irsERuDk7mjK+EKJpEugd2dH1xtWlhSdgyAMw+mlwdjdl6O3HzvDS6jQ2HsonyMuV347uzm0DwnF1kmAXwlIk0Du6qlJY/SdIecO4UfWNr0H4ANOG33ykgLnfpLE14zShvu48OLo7t/QPw9lRTsUKYTYJdGE4vA4+fxCKs+DqB2HkU8aZdhNordmQns+/Vqex60Qh3fw78cjYHkxKDMVRbqQhhGkk0MVPKoth9R9h+1sQ0MvYWw/rb9rwWmvWHsjlX9+ksf9kMVcFevDI2J7cEN9V7pAkhAkk0MWFDn1rrNZLTsLQR4we7E7mXThUX6/5et8pXlqTRlpOKb27ePHYuJ6Miw1GKQl2IVpLAl1cXGURfP0U7HwXAmPgxlchNMnUKerqNV/szmbemnSO5peREObDo+N6MrJnoAS7EK0ggS6al77aWK2X5sLQh4yz615dTJ2itq6ej3dm8e9v08k8U0H/CD9+N64nV3cPMHUeIeydBLq4tIoz8NVTsOs9UA4QPRISboPeE8HV07Rpqmvr+XD7CV759hCniisZEt2Z343vSXKkv2lzCGHPJNBFy+Wnw+7lxm3vCo+BcyfofYMR7tGjwNGcq0Ira+p4f+tx5q87TH5pFSN6BvLYuJ70Dfc1ZXwh7JUEurh8WsOJLUaw7/0YKgvBIxDiboGEWyEkCUzYAy+vruWdH47x2n8Pc6a8hrExwTw2riexId4mfBNC2B8JdNE2tdVwaLUR7ge/groq4wKlhNsg/lfgH9XmKUqranlz41EWbjhCSWUtN8R35dFxPege5GXCNyCE/ZBAF+apKITUz41tmYwNxtfCBxur9j43Qae27YUXldeweOMRlmw8SkVNHZMSQ3l4TA8iA9rWClgIeyGBLiyj8ATs+dBYuecdAAdn6DHeCPeeE9p0Ferpsmpe/+9h/u+HDGrqNJOTwnhwTHfC/NreNVIIWyaBLixLazi1xwj2PSug9BS4+kDsL41tmYih4NC6vi65JZW8uu4w7205jkZz+4BuPDCqO118zGlZIIStkUAXV059ndHhcfdyY2umuhS8wyDhV0a4B8W0atiTRRX879pDfLDtBA4Nt8W7T26LJzogCXRhHdXlcHCVsXI/9C3oOugSbwR73GTw7nrZQ544Xc6/v03nox2ZuDo5MikxhLExwQztHoC7i7TtFfZPAl1YX2ke7PvYCPes7YCC6BFGuMf8Alwv7zTLkbxS5q87zDf7TlFSVYurkwPDegQwJiaYMb2DCPKWLRlhnyTQRfuSfwj2NFy8dCYDnNyh9/VGuF81Ghxbfq/S6tp6th49zZrUHNak5pB5pgKAvmE+jI0JZmxsML27eEnfGGE3JNBF+6Q1ZG5ruHjpI6P9QKeAhouXbjMahV1GEGutOZhTwrepuazen8OuzEK0hlBfd8bEBDE2JphB0f5yRyVh0yTQRftXWw2H1jRcvPSlcfGS/1VGsCf8yrgv6mXKLalk3YFc1qTmsiE9j8qaejxcHBnRK5CxMcGM6hWEn4eLBb4ZISxHAl3Ylsoi2P+5Ee4ZGwENYQMbLl66GTw6X/6QNXVsOpzP6v25fJuaQ25JFQ4KkiP8jdV7bDBXBZrXhEwIS5FAF7arKNM42777A8jdDw5O0H2cEe69rmvVDa/r6zV7s4tYk5rLmv057D9ZDEBUgAdjY4IYExNMcoQfTnJPVNEOtTnQlVITgJcBR2Cx1vr5izznVuBZQAO7tNZ3NjemBLq4LFpDzt6fLl4qOQmu3hA1HEL7Q1gydE0Et8tv6pVVWMHa1BxWp+ay+XAB1XX1+Lg7M6pXIGNjgxneMxBvt5a/USuEJbUp0JVSjkAaMA7IBLYBd2it95/znB7AcmC01vqMUipIa53b3LgS6KLV6uuMPjJ7PoRjm+D0kYYHFAT2NgI+NMkI+aDYyzo1U1pVy4a0PNak5rL2QA5nymtwclAMju7c+MZquL+0HxDW09ZAHwI8q7W+tuHzPwBorf/fOc95AUjTWi9uaVES6MI05acha4dxvj0rxfi1vMB4zMkduvY9P+R9I1p0eqauXrPz+BlWp+bwbWouh3JLAegV7MXYWGNrJjHMV25+La6otgb6ZGCC1npmw+d3A4O01r895zmfYqzih2Jsyzyrtf7qImPNBmYDdOvWrf+xY8da9x0J0RytjfPtWdt/+ji5C2orjcc7BTQEfP+fgr4FXSIz8ssaz7tvyzhDXb0mwNOV0b2NUzPX9Aigk4s5NwARoilXItC/AGqAW4EwYD0Qr7UubGpcWaGLK6quBnL2nR/yeQcx3vLBOBYZmvxTyHeJb7ZbZFF5Dd+lGUcivzuYS0mlcbXq0O4BjVszwXK1qrCA5gK9JcuJLCD8nM/DGr52rkxgi9a6BjiqlEoDemDstwthfY7OEJJofAy4x/haZTFk7/wp4I+uN65gBaMVcJe480O+c/fGrpE+nZyZlBjKpMRQaurq2Xb0NKsbVu9rD+TyP5/sJT7UuFp1TEwQfUK85WpVYXEtWaE7YWynjMEI8m3AnVrrfec8ZwLGG6W/VkoFADuBRK11QVPjygpdtEtFWeev4rN3Gh0jwWgJHNrv/JD3Cj7v5Vpr0nNLWb0/h29Tc9h5wrhaNcTHjZG9g+gX7ktCmC/dgzxxlL130QpmHFu8HpiHsT++RGv9N6XUc0CK1vpzZSw9/gVMAOqAv2mtlzU3pgS6sAn1dcbWTGPIp0DOfqNzJIBPuLEHfzbkQxLB5ae7K+WXVrH2gHHefdPhAkqragFwd3YkLtSbhDBfEsJ8SAjzJbJzJ1nFi0uSC4uEMFN1ufEm67mnagqPG48pB+Oo5LkhH9gbHJ2or9ccyS9jT1Yhu04UsTuzkH3ZxVTV1gPg7eZEfEO49w3zIT7MlxAfNwl5cR4JdCEsrTTv/FV81najhQGAs0fD/n0/CO5jBH5gL3B2p7aunrScUiPkM42QP3CyhNp64//LAE8X4kN9zlvJB3rJTT06Mgl0Ia40raHg8Pkhf2oP1FUbjysH42RNUKzxEdzwq380lXVw4FQJezJ/CvlDuaU0ZDwhPm6NK/mEMB8SQn3x6SRXsnYUEuhCtAd1tXD6sNGTJme/8Wvufjh9lMbjk05uENCzYSUfA0F9IDiWMpdA9p0sYXdmIbsbQj6joLxx6MjOnYg/u1UT6kNcqA8ernIm3h5JoAvRnlWXQ96BhoBPNc7L56YaN9s+y823YTUf07iaL/LqwZ4Cxe6sQnY37MlnFxkXTzko6B7kSXyoL33DjZCP6eqNm7P0grd1EuhC2KLy0xeu5nNToar4p+d4hfy0XRMUyxnPHuyqDOTHU1WNK/n8UmObx8lB0burlxHyYT7Eh/nQM9gLZ+kqaVMk0IWwF1obLYVzUyF3X0PYp0L+wZ/tz18FQTHooFjOeHZnb20oWwp92JVVyu7MQoorjeOTrk4O9Ak59/ikD9EBntKfph2TQBfC3tXVGF0nz27XNLU/H9gLHRTDGY8eHNDhbCsL5vscF/aeLKa82jhb7+nqRJ8Qb3p38SI60JPoQA+iAz3p6u0mQd8OSKAL0VFVlxkXRp23P78fSnN+eo6bLzoohiKvnhxW4eysCmHt6QB25WnKGkIewM3ZgagAI+CvCvAgKtCD6IbPvaRf/BUjgS6EOF9ZwU8hn3t2VX/+/rx296PWoyulLoHkOwaQXefH4UpvUss82VPiQXa9H8V4AIpAL1eiA4yV/FWBHkQHehAV4Em4n7vc+clkEuhCiEtr3J9v2K4pPAHF2VCSbfxalnfBS2od3Y3Ad+hMVkPgZ9T4cEr7c0r7k+/QGQ+/LkQG+TSs7D0bwt4Dfw8XuQq2FSTQhRBtV1sFJaeM2/8VZxkhX9zw+5KTDeF/Euprz3tZHQ4UKH8y6/w4qf04pTtzSvtR7ByEo28IHgHd8O8aQVSwH9GBnkR07oSrkxyvbEpb2+cKIQQ4uYJfhPHRlPp6YyV/dlVfnI1jcTZBJScJLMoivjALVboXp9qGi6IKGz4OQb72Jkf7sVH7U+IaSJ1HCE6+IXgERtC5SwQh3boTFBggq/pmSKALIczj4GC0FPYKNnrXnEMBzmBs7VQVNwY+xdlUnclC5R2j85ksAkuzca88gldRERQB59zYrEy7cdopgHLXIGo9u+LsG4pnYAR+XSNx79wNfMLA3a9Ftxi0RxLoQogrSylw8zE+gmIAcG34OE9NJbrkJKdPHiU/O4PSvOPUFGbhWHIS98oc/Mt+ICinEKe0+vNeVqXcKHYJpsqjK9o7DGf/cDwCI/EMikD5hINPKDi7X5Fv9UqTQBdCtE/Obij/KDr7R9G5z4UPV9bUcTivhKzMDM6cyqD69HEoysK1LBvPyhyCKvIIKThAUMaFd8IsdfSh3K0LtV6hOPiE4RYYgVdgBI5+3YzA9+wCjrYXj7ZXsRBCAG7OjvQK8aVXSCKQeN5jWmuKK2rJLCxnV34RhTnHqMw/Rl3hCRxLsulUcRK/4jxCStIJObkZ74Pl572+DgfKXAKp9AhBe4Xg0rkbHoERuPhHGIHvE94ut3Yk0IUQdkcphU8nZ3w6+dAnxAfoBgw77zmVNXVkF1awq7CCnNw8SvMyqC44DsXGKt+rIoeuFQV0LUjB99jXuKjzT+/UKFfK3LtS6xmCg2847gHdcAvohvIJMwLfOxRcOl25bxoJdCFEB+Xm7NjQ2sATegQCsec9XlevySmuNEL/TBkFudlU5B+j7swJnEqy6FR5isCSPEJKcwnJ2Y/vwSKUOv8YeIWTz3l7+e4BETj6hkPXvhDQw/TvSQJdCCEuwtFBEeLrToivO8mR/kA4MKjxca01heU1ZBVW8GNhBScLiinJO05VwXFUcSauZSfxrcyha9VpQs4cJOT4JhyVsbXzY7dpJM542fSaJdCFEKIVlFL4ebjg5+FCXKgP0AXoed5zKqrryCqsaAz93Px8KvKP0697mEVqkkAXQggLcXdxpHuQJ92DPBu+0g1Isth80jVHCCHshAS6EELYCQl0IYSwExLoQghhJyTQhRDCTkigCyGEnZBAF0IIOyGBLoQQdsJqt6BTSuVxXuv6yxIA5JtYjqXZUr22VCvYVr22VCvYVr22VCu0rd4IrXXgxR6wWqC3hVIqpal76rVHtlSvLdUKtlWvLdUKtlWvLdUKlqtXtlyEEMJOSKALIYSdsNVAX2jtAi6TLdVrS7WCbdVrS7WCbdVrS7WCheq1yT10IYQQF7LVFboQQoifkUAXQgg7YXOBrpSaoJQ6qJQ6pJSaY+16mqOUWqKUylVK7bV2LZeilApXSq1TSu1XSu1TSj1s7ZqaopRyU0ptVUrtaqj1z9auqSWUUo5KqZ1KqS+sXUtzlFIZSqk9SqkflVIp1q7nUpRSvkqpFUqpA0qpVKXUEGvXdDFKqV4NP9OzH8VKqUdMncOW9tCVUo5AGjAOyAS2AXdorfdbtbAmKKWGA6XA21rrOGvX0xylVFegq9Z6h1LKC9gO3Ngef7ZKKQV4aK1LlVLOwEbgYa31ZiuX1iyl1GNAMuCttZ5o7XqaopTKAJK11jZxoY5S6v+ADVrrxUopF6CT1rrQ2nU1pyHLsoBBWuvWXmB5AVtboQ8EDmmtj2itq4FlwCQr19QkrfV64LS162gJrfVJrfWOht+XAKlAqHWrujhtKG341Lnho12vTJRSYcANwGJr12JPlFI+wHDgDQCtdXV7D/MGY4DDZoY52F6ghwInzvk8k3YaOrZMKRUJ9AO2WLeSpjVsX/wI5AKrtdbtttYG84AngHprF9ICGvhGKbVdKTXb2sVcQhSQB7zZsJ21WCnlYe2iWuB24H2zB7W1QBcWppTyBD4CHtFaF1u7nqZoreu01olAGDBQKdVut7SUUhOBXK31dmvX0kLXaK2TgOuABxq2DtsrJ4y7Li/QWvcDyoD2/t6aC/BL4EOzx7a1QM8Cws/5PKzha8IEDfvRHwFLtdYfW7uelmj45/U6YIK1a2nGUOCXDXvTy4DRSql3rVtS07TWWQ2/5gKfYGx1tleZQOY5/0JbgRHw7dl1wA6tdY7ZA9taoG8Deiilohr+lrsd+NzKNdmFhjca3wBStdZzrV1Pc5RSgUop34bfu2O8SX7AulU1TWv9B611mNY6EuO/2bVa67usXNZFKaU8Gt4Up2HrYjzQbk9paa1PASeUUr0avjQGaHdv5P/MHVhguwWMf67YDK11rVLqt8DXgCOwRGu9z8plNUkp9T4wEghQSmUCz2it37BuVU0aCtwN7GnYmwZ4Smu9yoo1NaUr8H8NJwUcgOVa63Z9FNCGBAOfGH+/4wS8p7X+yrolXdKDwNKGRd4RYLqV62lSw1+S44DfWGR8Wzq2KIQQomm2tuUihBCiCRLoQghhJyTQhRDCTkigCyGEnZBAF0IIOyGBLoQQdkICXQgh7MT/B1Gre37zuJOAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "def plot_history(history):\n",
    "    plt.plot(history.history['loss'],label=\"Training set loss\")\n",
    "    plt.plot(history.history['val_loss'],label=\"Validation set loss\")\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "plot_history(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Seems to be leveling out a bit, but improving still with no signs of overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prediction\n",
    "\n",
    "For prediction, we'll vectorize the given text normally, and initialize the decoder inputs to a vector with an initial start symbol and zeros otherwise.\n",
    "\n",
    "We'll then predict outputs, take the first new predicted character, place that back into the decoder inputs, and iterate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(text):\n",
    "    encoder_X = vectorize([text], input_tokenizer, INPUT_LENGTH, quiet=True)\n",
    "    decoder_X = np.zeros(shape=(1, OUTPUT_LENGTH))\n",
    "    decoder_X[0,0] = START_INDEX\n",
    "    predictions = []\n",
    "    for i in range(1, OUTPUT_LENGTH):\n",
    "        prediction = model.predict([encoder_X, decoder_X])[0][i].argmax()\n",
    "        predictions.append(prediction)\n",
    "        decoder_X[0,i] = prediction\n",
    "    pred_chars = []\n",
    "    for i in predictions:\n",
    "        if i == 0:\n",
    "            break\n",
    "        pred_chars += output_tokenizer.index_word[i]\n",
    "    return ''.join(pred_chars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test with a few cases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "James ジェームズ\n",
      "John ジョン\n",
      "Robert ロバート\n",
      "Mary マリー\n",
      "Patricia パトリキア\n",
      "Linda リンダ\n"
     ]
    }
   ],
   "source": [
    "test_inputs = ['James', 'John', 'Robert', 'Mary', 'Patricia', 'Linda']\n",
    "for text in test_inputs:\n",
    "    print(text, predict(text))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}