week8-recurrent-neural-networks/Recurrent Neural Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Assumptions in Machine Learning\n",
    "\n",
    "Every machine learning model has assumptions, indeed, that is the only way we can reasonably learn from data and make assumptions about how what we learn generalizes to the real world. \n",
    "\n",
    "The problem is the assumptions/difficulty tradeoff. Simpler models tend to have more naive assumptions (ie, assumptions that are not likely to be true or not likely to generalize), and more complex models (such as neural networks, convolutional networks, and recurrent neural networks) have less assumptions (and therefore, save for overfitting, are more likely to generalize better to the real world), difficult (and expensive) to build, train, and deploy. \n",
    "\n",
    "To understand this specifically, lets go through some models and state the assumptions they make. \n",
    "\n",
    "\n",
    "### Linear Regression & Logistic Regression\n",
    "\n",
    "Linear regression assumes that the data can be generated via a linear function, and logistic regression assumes that the data can be linearly separated. \n",
    "\n",
    "![lin](http://sebastianraschka.com/images/blog/2014/kernel_pca/linear_vs_nonlinear.png)\n",
    "\n",
    "Although we may intuitively think that assuming linearity is a very naive assumption, these are two of the most common algorithms used in the industry to make reasonable predictions for the following reasons: \n",
    "\n",
    "1. They are relatively easy to train, and don't require expensive & difficult research to optimize\n",
    "2. They have well known theoretical gaurantees and have been shown to work well in practice for decades\n",
    "3. Domain expertise can be applied to make sense of what has been learned\n",
    "4. The assumptions often turn out to be mostly true, and the tradeoff we have to make to get the edge cases classified correctly aren't worth the extra work.\n",
    "\n",
    "### Naive Bayes Classifier\n",
    "\n",
    "Bayes Rule states that for a latent variable and an evidence variable, the posterior distribution of the latent variable is equivalent to the prior probability of the latent variable times the likelihood of the evidence given the latent variable, divided by a constant (the probability of observing the evidence). \n",
    "\n",
    "Specifically:\n",
    "\n",
    "$$ p(y_i | x) = \\frac{p(y_i) p(x | y_i)}{p(x)} = \\frac{p(x, y_i)}{\\sum_i p(x, y_i)} $$\n",
    "\n",
    "Using the rules of probability, we can write the joint distribution as a conditional:\n",
    "\n",
    "$$ p(x, y_i) = p(x_1 | x_2, ... x_n, y_i) * p(x_2 | x_3, ... x_n, y_i) $$\n",
    "\n",
    "Here is where the assumption comes into play: since we can't really compute these conditional probabilities, we just define: \n",
    "\n",
    "$$ p(x_1 | x_2, ... x_n, y_i) = p(x_1 | y_i) $$\n",
    "\n",
    "This assumption means that each feature in our training set is entirely dependent upon the label that it is given, and is completely independent of every other feature, given the class label.\n",
    "\n",
    "We cal also set $ p(x_1 | y_i) $ to take on whatever probability distribution we want to (such as the Gaussian); doing this is also another assumption in our model. \n",
    "\n",
    "### Neural Networks\n",
    "- They also assume that the data are conditionally indpendent of each other. \n",
    "\n",
    "\n",
    "We have used Neural Networks and Convolutional Neural Netoworks to remove the assumptions of linearity in our data (and for convolutional neural networks, the assumption of spatial variance being important). Now we will use Recurrent Neural Networks to remove the assumption of independence (to an extent). \n",
    "\n",
    "Aside: A much simpler model (that doesn't work as well, but still has produced good results) that removes this independence assumption are Hidden Markov Models.\n",
    "\n",
    "\n",
    "\n",
    "### An Introduction to Recurrent Neural Networks in Tensorflow\n",
    "\n",
    "Recurrent Neural Networks are models that are able to model relationships between data where each input is not  independent of the previous inputs. \n",
    "\n",
    "For example, consider the field of Natural Language Processing (NLP). NLP deals with all tasks involved with processing texts and obtaining information (learning) from text. It is extremely common, especially today, to solve NLP tasks with machine learning (and especially deep learning). \n",
    "\n",
    "Recurrent Neural networks are a popular method for solving NLP problems, including predicting the sentiment of a movie review, predicting the next word (or character) in a sequence of words or characters, and even translating text to another language. This is because they can model dependencies, and often in text a current word is dependent on the previous words (you can probably think of several examples). \n",
    "\n",
    "Just for a break from all this technical talk, here is some Shakespeare: \n",
    "\n",
    "```\n",
    "PANDARUS:\n",
    "Alas, I think he shall be come approached and the day\n",
    "When little srain would be attain'd into being never fed,\n",
    "And who is but a chain and subjects of his death,\n",
    "I should not sleep.\n",
    "\n",
    "Second Senator:\n",
    "They are away this miseries, produced upon my soul,\n",
    "Breaking and strongly should be buried, when I perish\n",
    "The earth and thoughts of many states.\n",
    "\n",
    "DUKE VINCENTIO:\n",
    "Well, your wit is in the care of side and that.\n",
    "\n",
    "Second Lord:\n",
    "They would be ruled after this chamber, and\n",
    "my fair nues begun out of the fact, to be conveyed,\n",
    "Whose noble souls I'll have the heart of the wars.\n",
    "\n",
    "Clown:\n",
    "Come, sir, I will make did behold your worship.\n",
    "\n",
    "VIOLA:\n",
    "I'll drink it.\n",
    "```\n",
    "\n",
    "\n",
    "### The Power of Recurrent Neural Networks\n",
    "\n",
    "I lied, the above was not really Shakespeare. It was actually a poem written by a recurrent neural network that trained on a dataset of Shakespeare's writings. This should give you an idea of the power of RNNs: just by looking at characters (not even words!) it learned how to generate strikingly similar text to Shakespeare. Granted, it does not make much sense but it is likely that at least engineers will not be able to tell the difference.\n",
    "\n",
    "This was from an awesome article here http://karpathy.github.io/2015/05/21/rnn-effectiveness/\n",
    "\n",
    "\n",
    "Recurrent Neural Networks are important because they model sequential data. Each neuron, in addition to its data input, receives a previous ** state ** which is just a vector of numbers describing some previous computation in the RNN. This state input is what allows RNNs to learn dependencies in sequential data. \n",
    "\n",
    "It's a lot similar to you reading this sentence - when you understand the meaning of a particular word, you don't start figuring it out from scratch, but take the previous words and sentences into account. \n",
    "\n",
    "Here is what an RNN looks like: \n",
    "\n",
    "![rnn](https://cdn-images-1.medium.com/max/1600/1*V2W4TCmTj2h1CE7I-DngPw.png)\n",
    "\n",
    "\n",
    "And this is what an RNN looks like, predicting words character by character: \n",
    "\n",
    "![rnn-2](https://cdn-images-1.medium.com/max/1600/1*IMalbwl6uj3nlqxixZYFvA.jpeg)\n",
    "\n",
    "To understand RNNs deeply, we'll need to get into some data and code. \n",
    "\n",
    "### The data\n",
    "\n",
    "To clearly demonstrate the fundamental idea of RNNs learning dependencies in data, we will be using numerical data with clearly observable and quantifiable dependencies, and then we'll be able to check if the RNN can learn these dependencies. \n",
    "\n",
    "All of the code is written in a general enough format that words (represented as one-hot indices into a vocabulary or learned word vectors) can be fed into this model, however.\n",
    "\n",
    "\n",
    "**Input Sequence (X)** The input sequence is just a bunch of ones and zeros generated randomly. At each index, or more appropriately for RNNs, at each time *t* $X_t$ is 1 with probability 0.5 else 0. \n",
    "\n",
    "**Example:** $ X = [1, 0, 1, 1, 0, 0, ...] $\n",
    "\n",
    "**output Sequence (Y)** The output sequence will be generated from the input sequence, and our recurrent neural network will have to learn the rules by which the output sequence is generated. \n",
    "\n",
    "At each step in time $t$, $Y_t$ starts off as 1 with an initial probability of $0.5$. However if $X_{t-3} = 1$ then it $Y_t$ is almost surely 1 (ie, we raise the probability $p(y_t = 1)$ by 0.5 \n",
    "\n",
    "But, if $X_{t-8}$ is also 1 then we will decrease $P(y_t = 1)$ by 0.25. \n",
    "\n",
    "Hopefully, it is clear that if $X_{t-3} = 1$ and $X_{t-8} = 1$ then $y_t = 1 $ with probability 0.75. \n",
    "\n",
    "Note that there are no rules (ie, changes in $p(y_i = 1)$ if X is 0. \n",
    "\n",
    "This data is pretty simple. In fact, it is so simple that we can calculate exactly what our RNN should learn (given enough iterations and data) in order to determine if it has learned no dependencies, the dependency from 3 time steps ago, and the dependency from 8 time steps ago. \n",
    "\n",
    "#### Network Learns no dependencies\n",
    "\n",
    "To calculate the probability with which our network will predict $y_t = 1$, we can look at our generated data. \n",
    "\n",
    "First, in our generated data, $y_t = 1$ with initial probability 0.5. But we added 0.5 if $x_{t-3} = 1$, which itself has a probability $0.5$. So now, $p(y_t = 1) = 0.5 + 0.5^2 = 0.75$. However, we also need to remove .25 if $x_{t-8} = 1$, which has a probability of $0.5$. \n",
    "\n",
    "So in our generated data, $p(y_t = 1) = 0.5 + 0.5^2 - 0.25*0.5 = 0.625 $\n",
    "\n",
    "Since our network does not know any conditional dependence on our input and output data, it will just learn the distribution assuming that $y_t$ is independent of all $X$. So our network will assign every $y_t$ a probability of 0.625.\n",
    "\n",
    "The cross entropy loss that corresponds to this is $ -(.625 * log(.625) + (1 - .625) * log(1 - .625)) = 0.66$. \n",
    "\n",
    "#### Network Learns one dependency\n",
    "\n",
    "If our network realizes that whenever $x_{t-3} = 1$, the probability $p(y_t) = 1$ goes up by 0.5, then it will assign a probability of $0.875$ when $x_{t-3} = 1$, because $p(y_t = 1) = 0.625 + 0.5*.5 = 0.875$ and a probability of $0.625$ when $x_{t-3} = 1$. \n",
    "\n",
    "The corresponding cross entropy loss is $ -0.5 * (0.875 * log(0.875) + 0.125 * log(0.125)) + -0.5 * (0.625 * log(0.625) + 0.375 * log(0.375)))  = 0.52$\n",
    "\n",
    "#### Neural Network learns both dependencies\n",
    "\n",
    "Exercise: work out the probabilities with which our network will assign $p(y_t = 1)$, and calculate the cross-entropy. \n",
    "\n",
    "To check your answer, the cross entropy you should get is 0.45. \n",
    "\n",
    "\n",
    "So now we know that if we train our network correctly: \n",
    "\n",
    "- With one hidden state, we should learn no dependencies, so our final loss should be 0.66. \n",
    "\n",
    "- With three or more hidden states, we should learn the first dependency, so our final loss should be 0.52. \n",
    "\n",
    "- With eight or more hidden states, we should learn both dependencies, so our final loss should be 0.46. \n",
    "\n",
    "Let's start building our RNN to see if our hypothesis is supported. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import tensorflow as tf\n",
    "from gen_data import gen_data # gets the data for this project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "RNNconfig = {\n",
    "    'num_steps' : 5, # higher n = capture longer term dependencies, but more expensive (and potential vanishing gradient issues)\n",
    "    'batch_size' : 200,\n",
    "    'state_size' :4,\n",
    "    'learning_rate' : 0.1\n",
    "}\n",
    "\n",
    "num_classes = 2\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# adapted from https://github.com/tensorflow/tensorflow/blob/master/tensorflow/models/rnn/ptb/reader.py\n",
    "def gen_batch(raw_data, batch_size, num_steps):\n",
    "    raw_x, raw_y = raw_data\n",
    "    data_length = len(raw_x)\n",
    "\n",
    "    # partition raw data into batches and stack them vertically in a data matrix\n",
    "    batch_partition_length = data_length // batch_size\n",
    "    data_x = np.zeros([batch_size, batch_partition_length], dtype=np.int32)\n",
    "    data_y = np.zeros([batch_size, batch_partition_length], dtype=np.int32)\n",
    "    for i in range(batch_size):\n",
    "        data_x[i] = raw_x[batch_partition_length * i:batch_partition_length * (i + 1)]\n",
    "        data_y[i] = raw_y[batch_partition_length * i:batch_partition_length * (i + 1)]\n",
    "    # further divide batch partitions into num_steps for truncated backprop\n",
    "    epoch_size = batch_partition_length // num_steps\n",
    "\n",
    "    for i in range(epoch_size):\n",
    "        x = data_x[:, i * num_steps:(i + 1) * num_steps]\n",
    "        y = data_y[:, i * num_steps:(i + 1) * num_steps]\n",
    "        yield (x, y)\n",
    "\n",
    "def gen_epochs(n, num_steps):\n",
    "    for i in range(n):\n",
    "        yield gen_batch(gen_data(), RNNconfig['batch_size'], num_steps) # python trivia: why do we use yield instead of return here?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "\n",
    "# placeholder\n",
    "x = tf.placeholder(tf.int32, [RNNconfig['batch_size'], RNNconfig['num_steps']], name='input_placeholder')\n",
    "y = tf.placeholder(tf.int32, [RNNconfig['batch_size'], RNNconfig['num_steps']], name='labels_placeholder')\n",
    "init_state = tf.zeros([RNNconfig['batch_size'], RNNconfig['state_size']])\n",
    "\n",
    "\"\"\"\n",
    "RNN Inputs\n",
    "\"\"\"\n",
    "\n",
    "# Turn our x placeholder into a list of one-hot\n",
    "# rnn_inputs is a list of num_steps tensors with shape [batch_size, num_classes]\n",
    "x_one_hot = tf.one_hot(x, num_classes) # note: num_classes is not an RNN variable...\n",
    "rnn_inputs = tf.unstack(x_one_hot, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nfrom __call__ method\\nhttps://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/rnn/python/ops/rnn_cell.py\\n'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"\n",
    "from __call__ method\n",
    "https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/rnn/python/ops/rnn_cell.py\n",
    "\"\"\"\n",
    "# with tf.variable_scope('rnn_cell'):\n",
    "#     W = tf.get_variable('W', [num_classes + RNNconfig['state_size'], RNNconfig['state_size']])\n",
    "#     b = tf.get_variable('b', [RNNconfig['state_size']], initializer=tf.constant_initializer(0.0))\n",
    "\n",
    "# def rnn_cell(rnn_input, state):\n",
    "#     with tf.variable_scope('rnn_cell', reuse=True):\n",
    "#         W = tf.get_variable('W', [num_classes + RNNconfig['state_size'], RNNconfig['state_size']])\n",
    "#         b = tf.get_variable('b', [RNNconfig['state_size']], initializer=tf.constant_initializer(0.0))\n",
    "#     return tf.tanh(tf.matmul(tf.concat([rnn_input, state], 1), W) + b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/rnn/python/ops/rnn_cell.py\n",
    "\"\"\"\n",
    "# state = init_state\n",
    "# rnn_outputs = []\n",
    "# for rnn_input in rnn_inputs:\n",
    "#     state = rnn_cell(rnn_input, state)\n",
    "#     rnn_outputs.append(state)\n",
    "# final_state = rnn_outputs[-1]\n",
    "cell = tf.contrib.rnn.BasicRNNCell(RNNconfig['state_size'])\n",
    "\n",
    "rnn_outputs, final_state = tf.contrib.rnn.static_rnn(cell, rnn_inputs, initial_state = init_state)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/seq2seq/python/ops/loss.py#L30\n",
    "\"\"\"\n",
    "\n",
    "#logits and predictions\n",
    "with tf.variable_scope('softmax'):\n",
    "    W = tf.get_variable('W', [RNNconfig['state_size'], num_classes])\n",
    "    b = tf.get_variable('b', [num_classes], initializer=tf.constant_initializer(0.0))\n",
    "logits = [tf.matmul(rnn_output, W) + b for rnn_output in rnn_outputs]\n",
    "predictions = [tf.nn.softmax(logit) for logit in logits]\n",
    "\n",
    "# Turn our y placeholder into a list of labels\n",
    "y_as_list = tf.unstack(y, num=RNNconfig['num_steps'], axis=1)\n",
    "\n",
    "#losses and train_step\n",
    "losses = [tf.nn.sparse_softmax_cross_entropy_with_logits(labels=label, logits=logit) for \\\n",
    "          logit, label in zip(logits, y_as_list)]\n",
    "total_loss = tf.reduce_mean(losses)\n",
    "train_step = tf.train.AdagradOptimizer(0.1).minimize(total_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "\n",
    "def train_network(num_epochs, verbose=True):\n",
    "    \n",
    "    with tf.Session() as sess:\n",
    "        sess.run(tf.global_variables_initializer())\n",
    "        training_losses = []\n",
    "        for idx, epoch in enumerate(gen_epochs(num_epochs, RNNconfig['num_steps'])):\n",
    "            training_loss = 0\n",
    "            training_state = np.zeros((RNNconfig['batch_size'], RNNconfig['state_size']))\n",
    "            if verbose:\n",
    "                print(\"\\nEPOCH\", idx)\n",
    "            for step, (X, Y) in enumerate(epoch):\n",
    "                tr_losses, training_loss_, training_state, _ = \\\n",
    "                    sess.run([losses,\n",
    "                              total_loss,\n",
    "                              final_state,\n",
    "                              train_step],\n",
    "                                  feed_dict={x:X, y:Y, init_state:training_state})\n",
    "                training_loss += training_loss_\n",
    "                if step % 100 == 0 and step > 0:\n",
    "                    if verbose:\n",
    "                        print(\"Average loss at step\", step,\n",
    "                              \"for last 100 steps:\", training_loss/100)\n",
    "                    training_losses.append(training_loss/100)\n",
    "                    training_loss = 0\n",
    "\n",
    "    return training_losses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "EPOCH 0\n",
      "Average loss at step 100 for last 100 steps: 0.649515368342\n",
      "Average loss at step 200 for last 100 steps: 0.586091864705\n",
      "Average loss at step 300 for last 100 steps: 0.525080781877\n",
      "Average loss at step 400 for last 100 steps: 0.52101107955\n",
      "Average loss at step 500 for last 100 steps: 0.520254173279\n",
      "Average loss at step 600 for last 100 steps: 0.519741330445\n",
      "Average loss at step 700 for last 100 steps: 0.520542836189\n",
      "Average loss at step 800 for last 100 steps: 0.518190928996\n",
      "Average loss at step 900 for last 100 steps: 0.52299423635\n",
      "\n",
      "EPOCH 1\n",
      "Average loss at step 100 for last 100 steps: 0.523963079751\n",
      "Average loss at step 200 for last 100 steps: 0.517871082425\n",
      "Average loss at step 300 for last 100 steps: 0.521744145155\n",
      "Average loss at step 400 for last 100 steps: 0.517919010222\n",
      "Average loss at step 500 for last 100 steps: 0.518859588206\n",
      "Average loss at step 600 for last 100 steps: 0.52037773937\n",
      "Average loss at step 700 for last 100 steps: 0.518258718252\n",
      "Average loss at step 800 for last 100 steps: 0.518916441202\n",
      "Average loss at step 900 for last 100 steps: 0.519827694595\n",
      "\n",
      "EPOCH 2\n",
      "Average loss at step 100 for last 100 steps: 0.526057345569\n",
      "Average loss at step 200 for last 100 steps: 0.518300784528\n",
      "Average loss at step 300 for last 100 steps: 0.517992891669\n",
      "Average loss at step 400 for last 100 steps: 0.519173080623\n",
      "Average loss at step 500 for last 100 steps: 0.519833306074\n",
      "Average loss at step 600 for last 100 steps: 0.518802602589\n",
      "Average loss at step 700 for last 100 steps: 0.519270105958\n",
      "Average loss at step 800 for last 100 steps: 0.521131120324\n",
      "Average loss at step 900 for last 100 steps: 0.516737457812\n",
      "\n",
      "EPOCH 3\n",
      "Average loss at step 100 for last 100 steps: 0.526474792957\n",
      "Average loss at step 200 for last 100 steps: 0.518475809693\n",
      "Average loss at step 300 for last 100 steps: 0.518512052596\n",
      "Average loss at step 400 for last 100 steps: 0.519411894977\n",
      "Average loss at step 500 for last 100 steps: 0.519316830337\n",
      "Average loss at step 600 for last 100 steps: 0.519966934025\n",
      "Average loss at step 700 for last 100 steps: 0.519472878277\n",
      "Average loss at step 800 for last 100 steps: 0.522138119638\n",
      "Average loss at step 900 for last 100 steps: 0.520412448347\n",
      "\n",
      "EPOCH 4\n",
      "Average loss at step 100 for last 100 steps: 0.52410581857\n",
      "Average loss at step 200 for last 100 steps: 0.517132446766\n",
      "Average loss at step 300 for last 100 steps: 0.520943225026\n",
      "Average loss at step 400 for last 100 steps: 0.52049028337\n",
      "Average loss at step 500 for last 100 steps: 0.517826253176\n",
      "Average loss at step 600 for last 100 steps: 0.518392149508\n",
      "Average loss at step 700 for last 100 steps: 0.521618328989\n",
      "Average loss at step 800 for last 100 steps: 0.520652165711\n",
      "Average loss at step 900 for last 100 steps: 0.520495499671\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x112ad3a20>]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4lPW5//H3nWSGZAJZgLAFEEQQd4RIXWqltrbYerCe\nWuta8bTa1vrrals8pz97jl49Paer/VVaq9aqta1trbVosUhbrEtdCAjIvikQCCQQsu+Z+/fHTOIQ\nEjJAYGCez+u65so8zzwzc88D+cw33+eZe8zdERGRYMhIdQEiInL0KPRFRAJEoS8iEiAKfRGRAFHo\ni4gEiEJfRCRAFPoiIgGi0BcRCRCFvohIgGSluoDuhg4d6uPGjUt1GSIix5UlS5bsdveivrY75kJ/\n3LhxlJaWproMEZHjipltSWY7Te+IiASIQl9EJEAU+iIiAaLQFxEJEIW+iEiAKPRFRAJEoS8iEiBp\nE/p1zW38cOF6lm2rTnUpIiLHrLQJ/WgUfvS3DZS+XZXqUkREjllpE/qDsrPIMKhpakt1KSIix6y0\nCf2MDCM/J8TextZUlyIicsxKm9AHKIyE2duokb6ISG/SKvQLIiGqNdIXEelVWoV+YSRMtUb6IiK9\nSqvQz4+EFPoiIgeQVqEfm9PX9I6ISG/SLPRDNLZ20NLekepSRESOSWkV+gWRMAA1muIREelRmoV+\nCECnbYqI9CKp0DezmWa2zsw2mtmcXra5ysxWm9kqM/t1t9vyzKzMzO7tj6J7Uxgf6WteX0SkZ31+\nMbqZZQJzgUuAMmCxmc1z99UJ20wE7gAucPe9Zjas28PcDbzQf2X3rHOkr3P1RUR6lsxIfzqw0d03\nu3sr8DhwebdtbgbmuvteAHev6LzBzKYBw4Hn+qfk3nWO9HXapohIz5IJ/WJgW8JyWXxdoknAJDN7\n2cxeNbOZAGaWAXwfuP1AT2Bmt5hZqZmVVlZWJl99N5rTFxE5sP46kJsFTARmANcAD5hZAXArMN/d\nyw50Z3e/391L3L2kqKjokIvICWUSzsrQ9I6ISC/6nNMHtgNjEpZHx9clKgNec/c24C0zW0/sTeA8\n4EIzuxUYCITNrN7dezwYfLjMjMKIOm2KiPQmmZH+YmCimY03szBwNTCv2zZPERvlY2ZDiU33bHb3\n69x9rLuPIzbF8+iRCvxO6r8jItK7PkPf3duB24AFwBrgd+6+yszuMrNZ8c0WAHvMbDWwCPiqu+85\nUkUfSH6O+u+IiPQmmekd3H0+ML/bujsTrjvw5filt8d4GHj4UIo8GIWRMJsq64/004iIHJfS6hO5\nAIW5IZ29IyLSi7QL/YJImJqmVmJ/fIiISKL0C/2cEG0dTkOrOm2KiHSXdqHf1X+nQadtioh0l3ah\n/07/Hc3ri4h0l3ahX5gb77/TpJG+iEh3aRf6BTnqvyMi0pv0C/2uTpsa6YuIdJeGoR8f6TdopC8i\n0l3ahX4oM4NBA7I0py8i0oO0C32A/Ij674iI9CQtQ78wElZ7ZRGRHqRl6BdE1H9HRKQnaRn6hZEw\nNRrpi4jsJy1DXyN9EZGepWnoh6ltbqMjqk6bIiKJ0jL0CyMh3KGmSaN9EZFEaRn67zRd07y+iEii\nNA39eHtlzeuLiOwjLUO/UP13RER6lKahr06bIiI9SSr0zWymma0zs41mNqeXba4ys9VmtsrMfh1f\nN8XMXomvW2FmH+/P4ntTkKORvohIT7L62sDMMoG5wCVAGbDYzOa5++qEbSYCdwAXuPteMxsWv6kR\n+IS7bzCzUcASM1vg7tX9/koSDMrOIsP07VkiIt0lM9KfDmx0983u3go8DlzebZubgbnuvhfA3Svi\nP9e7+4b49R1ABVDUX8X3JiPDKFD/HRGR/SQT+sXAtoTlsvi6RJOASWb2spm9amYzuz+ImU0HwsCm\nHm67xcxKzay0srIy+eoPoECdNkVE9tNfB3KzgInADOAa4AEzK+i80cxGAr8EbnL3aPc7u/v97l7i\n7iVFRf3zh0BBTkg99UVEukkm9LcDYxKWR8fXJSoD5rl7m7u/Bawn9iaAmeUBfwb+w91fPfySk1MY\nCevbs0REukkm9BcDE81svJmFgauBed22eYrYKB8zG0psumdzfPs/Ao+6+xP9VnUSCiJhnb0jItJN\nn6Hv7u3AbcACYA3wO3dfZWZ3mdms+GYLgD1mthpYBHzV3fcAVwHvAWab2bL4ZcoReSXdFKrTpojI\nfvo8ZRPA3ecD87utuzPhugNfjl8St3kMeOzwyzx4BZEQTW0dNLd1kB3KTEUJIiLHnLT8RC68039H\nnTZFRN6RtqFf2NV0TfP6IiKd0jj0O9sra6QvItIpbUM/Xz31RUT2k7ahX6ie+iIi+wlA6GukLyLS\nKW1DPyecyYCsDGo00hcR6ZK2oQ+xc/U10hcReUdah35hJKw5fRGRBGkd+rH2yhrpi4h0SuvQL4yE\ndZ6+iEiCtA79AjVdExHZR5qHfqy9cqwfnIiIpHXoF0ZCtEed+pb2VJciInJMSOvQ7+y0qXl9EZGY\n9A79HDVdExFJlNahX5irVgwiIonSO/TjnTYV+iIiMWkd+vr2LBGRfaV16OfH5/T3Nij0RUQgydA3\ns5lmts7MNprZnF62ucrMVpvZKjP7dcL6G81sQ/xyY38VnoxQZgaDBmRpekdEJC6rrw3MLBOYC1wC\nlAGLzWyeu69O2GYicAdwgbvvNbNh8fWDgW8CJYADS+L33dv/L6VnBbnqvyMi0imZkf50YKO7b3b3\nVuBx4PJu29wMzO0Mc3eviK//ILDQ3avity0EZvZP6ckpjISp1py+iAiQXOgXA9sSlsvi6xJNAiaZ\n2ctm9qqZzTyI+x5R+TnqvyMi0qnP6Z2DeJyJwAxgNPCCmZ2R7J3N7BbgFoCxY8f2U0kxhZEwW6sa\n+/UxRUSOV8mM9LcDYxKWR8fXJSoD5rl7m7u/Bawn9iaQzH1x9/vdvcTdS4qKig6m/j4VRkLsbdCc\nvogIJBf6i4GJZjbezMLA1cC8bts8RWyUj5kNJTbdsxlYAHzAzArNrBD4QHzdUVMQCVPb3E5HVJ02\nRUT6nN5x93Yzu41YWGcCD7n7KjO7Cyh193m8E+6rgQ7gq+6+B8DM7ib2xgFwl7tXHYkX0puC+Kdy\na5raGBxvyyAiElRJzem7+3xgfrd1dyZcd+DL8Uv3+z4EPHR4ZR66wsg7/XcU+iISdGn9iVx4Z6Sv\nc/VFRAIQ+oXqqS8i0iXtQ7+gq9OmQl9EJACh3znS1/SOiEjah35edhaZGaamayIiBCD0zYyCnJDm\n9EVECEDoA+RHFPoiIhCQ0C+MhDW9IyJCYEJfnTZFRCAgoZ+fE6ZGI30RkWCEvkb6IiIxwQj93DBN\nbR00t3WkuhQRkZQKROi/039Ho30RCbZghH5O/FO5TZrXF5FgC0ToF3b232nQSF9Egi0Qoa/+OyIi\nMYEI/cJcddoUEYGghH7Ct2eJiARZIEI/O5RJTiiTvQ0KfREJtkCEPsDg3DBVGumLSMAFJvQL1GlT\nRCS50DezmWa2zsw2mtmcHm6fbWaVZrYsfvlUwm3fMbNVZrbGzP6fmVl/voBkDc4NU6XpHREJuKy+\nNjCzTGAucAlQBiw2s3nuvrrbpr9199u63fd84ALgzPiql4CLgOcPs+6DVhAJs62q8Wg/rYjIMSWZ\nkf50YKO7b3b3VuBx4PIkH9+BbCAMDABCwK5DKfRwDY6ENNIXkcBLJvSLgW0Jy2Xxdd191MxWmNkT\nZjYGwN1fARYB5fHLAndfc5g1H5KCSJja5nbaO6KpeHoRkWNCfx3IfRoY5+5nAguBRwDM7CTgFGA0\nsTeKi83swu53NrNbzKzUzEorKyv7qaR9Dc7t7L+jg7kiElzJhP52YEzC8uj4ui7uvsfdW+KLDwLT\n4tevAF5193p3rweeBc7r/gTufr+7l7h7SVFR0cG+hqS802lTUzwiElzJhP5iYKKZjTezMHA1MC9x\nAzMbmbA4C+icwtkKXGRmWWYWInYQNyXTO50j/So1XRORAOvz7B13bzez24AFQCbwkLuvMrO7gFJ3\nnwd83sxmAe1AFTA7fvcngIuBN4kd1P2Luz/d/y+jb2rFICKSROgDuPt8YH63dXcmXL8DuKOH+3UA\nnz7MGvtFYXykr1YMIhJkgflEbldPfX0qV0QCLDChnxPKZEBWhqZ3RCTQAhP6ZkZhJKzpHREJtMCE\nPsTm9TXSF5EgC1boR0Ka0xeRQAtW6OdqekdEgi1YoR8JaXpHRAItUKE/OBKmuqmNjqinuhQRkZQI\nVOgXRMK4Q62arolIQAUq9Lv672iKR0QCKlChr06bIhJ0gQp9ddoUkaALVOir06aIBF2wQl+dNkUk\n4AIV+rnhTEKZpk/likhgBSr01XRNRIIuUKEPsXl9zemLSFAFL/Rz1YpBRIIreKEfCWtOX0QCK3ih\nr06bIhJgwQv9SIjqpjaiaromIgGUVOib2UwzW2dmG81sTg+3zzazSjNbFr98KuG2sWb2nJmtMbPV\nZjau/8o/eIWRMB1Rp665PZVliIikRFZfG5hZJjAXuAQoAxab2Tx3X91t09+6+209PMSjwLfcfaGZ\nDQSih1v04Uj8VG5+vBePiEhQJDPSnw5sdPfN7t4KPA5cnsyDm9mpQJa7LwRw93p3bzzkavuBOm2K\nSJAlE/rFwLaE5bL4uu4+amYrzOwJMxsTXzcJqDazJ83sDTP7bvwvh5RRp00RCbL+OpD7NDDO3c8E\nFgKPxNdnARcCtwPnACcCs7vf2cxuMbNSMyutrKzsp5J6pk6bIhJkyYT+dmBMwvLo+Lou7r7H3Vvi\niw8C0+LXy4Bl8amhduApYGr3J3D3+929xN1LioqKDvY1HJSC+Jy+RvoiEkTJhP5iYKKZjTezMHA1\nMC9xAzMbmbA4C1iTcN8CM+tM8ouB7geAj6q87CwyM4wqnasvIgHU59k77t5uZrcBC4BM4CF3X2Vm\ndwGl7j4P+LyZzQLagSriUzju3mFmtwN/MzMDlgAPHJmXkpxY07WQPpUrIoHUZ+gDuPt8YH63dXcm\nXL8DuKOX+y4EzjyMGvudOm2KSFAF7hO5oE6bIhJcwQx9ddoUkYAKZuir06aIBFQwQz/eadNdTddE\nJFiCGfqREO1Rp75FTddEJFgCGvrxpmv6VK6IBEywQ18Hc0UkYIIZ+rkKfREJpmCGfrzTpkJfRIIm\nkKHf2WlTc/oiEjSBDP287BAZppG+iARPIEM/I8MoUCsGEQmgQIY+xL5BS9M7IhI0gQ39wRrpi0gA\nBTb0CyJhfZGKiAROYEN/cG6IajVdE5GACWzoF0bCVDWq6ZqIBEtwQz83TGt7lKa2jlSXIiJy1AQ3\n9OOfytW8vogESYBDP/apXM3ri0iQBDf0460YNNIXkSBJKvTNbKaZrTOzjWY2p4fbZ5tZpZkti18+\n1e32PDMrM7N7+6vww6X2yiISRFl9bWBmmcBc4BKgDFhsZvPcfXW3TX/r7rf18jB3Ay8cVqX9rKvT\npkb6IhIgyYz0pwMb3X2zu7cCjwOXJ/sEZjYNGA48d2glHhn5OSHM0Beki0igJBP6xcC2hOWy+Lru\nPmpmK8zsCTMbA2BmGcD3gdsP9ARmdouZlZpZaWVlZZKlH56szAzyskOa3hGRQOmvA7lPA+Pc/Uxg\nIfBIfP2twHx3LzvQnd39fncvcfeSoqKifiqpb4Nzwxrpi0ig9DmnD2wHxiQsj46v6+LuexIWHwS+\nE79+HnChmd0KDATCZlbv7vsdDE6FWKdNjfRFJDiSCf3FwEQzG08s7K8Grk3cwMxGunt5fHEWsAbA\n3a9L2GY2UHKsBD7EOm3urG1OdRkiIkdNn6Hv7u1mdhuwAMgEHnL3VWZ2F1Dq7vOAz5vZLKAdqAJm\nH8Ga+01BJMya8tpUlyEictQkM9LH3ecD87utuzPh+h3AHX08xsPAwwdd4RE0ODekOX0RCZTAfiIX\nYiP9prYOmtV0TUQCItChPzhXn8oVkWAJdOir06aIBE3AQ1+dNkUkWIId+uq0KSIBE+zQ7xrpK/RF\nJBgCHfoFXXP6mt4RkWAIdOiHMjMYlJ2ls3dEJDACHfoQm+JR6ItIUCj01WlTRAJEoa9OmyISIIEP\n/cGa3hGRAAl86BdEwhrpi0hgBD70B+eGaGjtoKVdTdfk+NARdUrfrsLdU12KHIcCH/oFasVw3PrH\n+kru+ev6wIXfo6+8zZX3vcK85TtSXYochwIf+uq0eXzaVFnPZx9bwj1/3cCzK3emupyjprU9ys/+\nsRmAb89fS2Nre4orkuNN4EN/VEEOAD/7x2Za26MprkaS0dzWwed+tZQBWRlMGj6Qu59ZTUNLMMLv\nj2+UsbO2mS+8byI7a5v56fObUl2SHGcCH/pnjc7nS++fxB/f2M6/PbyY2ubjZ5pnZ00zC1bt5PvP\nreMf6ytTXc5Rc9czq1m7s44ffHwK3/7XMyivaebeRRtTXdYR1xF1fvr8Js4cnc8X3z+Ry6eM4mcv\nbGZbVWOqSzsqtuxp4JZHS/nnpt2pLuW4ltTXJaYzM+ML759IcWEOc/6wgqvue4Vf3HQOI/NzUl3a\nPppaOyjdUsXybdUsL6thRVk1u2pbum7PCWXy1Ocu4OQRg1JY5ZH39PId/Pq1rXz6ohN578nDAPjo\n1NE8+OJmrpw2mglFA1Nc4ZHz5zfLeXtPI/ddPw0zY86lk3lu1S6+9ec13HfDtFSXd0Q9v66Cz//m\nDWqb21m2rZqFX7qI/HjvLDk4gR/pd7py2mgevmk6ZXubuGLuP1m949j5wvSKumY+/OMXueHnr/O9\n59azqaKe8ycM5Zv/cipP3no+L37tveQOyOKzv1pCfRpPc7y9u4E7nnyTqWMLuP0DJ3etn3PpZLJD\nmfznvFVpe1DX3fnJoo2cNGwgHzh1OAAj83O4dcYE/rJqJ//cmJ6jX3dn7qKN3PTwYooLI/z0uqns\naWjlv55ZlerSjltJhb6ZzTSzdWa20czm9HD7bDOrNLNl8cun4uunmNkrZrbKzFaY2cf7+wX0p3dP\nHMrvP3MeAFf97BVeOAamTPbUt3DdA6+xs6aZn1w3leXf/AB/v30GP/z4FG66YDxTxxYyZnCEH19z\ndlcoJht8G3bVHTdz4S3tHdz2m6VkZhg/vnYqocx3/usWDRrAVy6ZxIsbdvOXND2o+/e1FazdWcet\nMyaQkWFd629+z4mMLszhv55eTXvH8XFMaunWvdzw89e4+dFS/rKyvNdjafUt7XzmsSV8d8E6Zp01\niic/ez6XnjGSW2dM4Mml2/nbml1HufLkuDtNrcfuKeDWV0CYWSawHrgEKAMWA9e4++qEbWYDJe5+\nW7f7TgLc3TeY2ShgCXCKu1f39nwlJSVeWlp6iC+nf5TXNHHTLxazsaKeuy4/nSvOLiYnnHnU66hu\nbOXaB15jU2U9D980nfMmDDng9nMXbeS7C9Zx9+WnccN543rdzt356T828d0F6xg/NJef33gO44fm\n9nP1/eubf1rJI69s4YFPlHBJfKSbqL0jymU/fonapjb++pWLiISP/ZnLhpZ2/rlpD+8+aegB/3+5\nO//6039SWdfCottn7POGB/CXleV85rGl3HX5aXziAP/uqVZe08T/PruWp5btoGjQANxhd30LhZEQ\ns84axUenjeaM4nzMjE2V9Xz6l0t4a3cDd1w6mU++ezxmsTe7lvYOZv34ZfY2tiY9zdPY2k44M4Os\nzCM7ubFlTwNf+d1ySrfsZfzQXM4anc+Zows4a0w+p43KJzt05HLEzJa4e0mf2yUR+ucB/+nuH4wv\n3wHg7t9O2GY2PYR+D4+1HLjS3Tf0ts2xEPoAdc1t3Pqrpby4YTdmUFyQw8RhA5k4fBAnDRvIScMG\nMqFoIPk5yc0r1jW3sX5XPet21pGVacw6a9QB/wPUNrdxw4Ovsaa8jgduLOGiSUV9Pkc06nzykcW8\ntHE3T3zmfM4aU7DfNs1tHfz7k2/y5Bvbed/kYSzdupeOqDP3uqlcOLHv5ziQlvYONlU0MGn4wH79\n5Xr2zXI++6ulfPLd4/m/l53a63aL367iY/e9wq0zJvC1mZP77fn728rtNfz69a386Y3tNLR2UHJC\nIT+ffU6v/5de2bSHax54lbs/cjo3nHvCfre7O9c+8Bqry2t5/vYZXd8Id6xoau3g/hc2c98/NtHh\nzs0XjufWGScxICuDFzfs5omlZSxcvYvW9igThw3k4lOG8etXtxLKyuDea8/m/AlD93vMN8tq+MhP\nXuYjU4r5/lVn9frc7s5Pnt/E959bhwNDcsMUDcpm2KABsUveAMYURpg1ZdRhDRTcnccXb+PuZ1aT\nmWFcO30sm3c37HPsLTPDOHn4IE4syiU7lEl2KIPsrEwGxH9mhzIZWZDNZWeOOqQa+jP0rwRmunvn\nlM0NwLsSAz4e+t8GKon9VfAld9/W7XGmA48Ap7l7r3+HHiuhD9DWEeXvaytYt7OODRX1bKyoZ1Nl\n/T5/jg4ckMWI/GxG5mczIi/2c2RBDqHMDDZU1LF+Zx3rd9Wzvbppn8cekZfN5947gavOGcOArH3D\nv6GlnU889DrLt1Vz3/XTeH8PI9ve7G1o5bIfvwTAnz//7q4PnwFU1rXw6V+WsnRrNV++ZBL/5+KT\nKNvbxKceKWVjZT3f+PApzD5/XNeIKhnNbR28sL6SZ1fu5K+rd1HX0s6YwTl86t0n8rGS0Yf8i+Tu\n7KptYdm2ar76xHJOLBrI7z99HuGsA7+ZfPl3y3h6+Q4WfPE9nNgPB3WjUae8tpktuxvYUtVIYSTE\n1LGFDMvLPqjHaWhpZ97yHfzm9a2sKKthQFYGl505ilNGDuJ//7KWScMH8ei/TWfIwAH73ff6B19j\n3a46Xvzae3sdKKzdWcuHfvQi1597AnddfvohvdZkNLd1ULa3kS17Gnl7TyNb9jTQ1uFd//eHx38P\nRuRlk5eTxdMryvmf+WvYUdPMh88YyZxLJzNmcGS/x61pbOOZN3fwhyVlLN1azRnF+dx3wzSKC3o/\noeJ7C9Zx76KNPDS7hIsn7/870tTawVefWM4zK8q59PQRTBo+iIq6Firrmqmoa6GitoXK+hY6orH6\nv37pyVx+VvE+02fJqKxrYc4fVvC3tRVccNIQvnvlWV2ngkPsLLvlZdWsKKtm+bYayvY20tIepbmt\no+tnNB7DZ48t4I+3XnBQz9/paIf+EKDe3VvM7NPAx9394oTbRwLPAze6+6s9PMctwC0AY8eOnbZl\ny5YkXmJqdESdbVWNbKio563d9ZTXNFNe3Ux5bTM7a5qoqGuhc5eGMo0JRQM5ecQgJg0fxOT4z21V\njfxg4XpKt+xlVH42n7v4JD42bQzhrAyaWju46eHXef2tKu69diofOmPkQde4bFs1H7vvn7xnYhEP\nfKKEjAxj9Y5abn60lD0NLfzwqilcmvC49S3tfOm3y1i4ehcfLxnD3R85/YDh2tjazqK1lTy7spy/\nr62gsbWDgkiID5w6nLPGFHT94hZEQtxw7gnceP44hvYQZp2iUWd7dROrdtSwcnstK+M/d9fHRkhD\ncsM89bkLegyL7irrWrj4e88zZWwBj/7b9IN6A9tW1cirm/ewbmcdb+9p5O09DWytauxxzrm4IIez\nxxYwdWwhU08o5NSReWRlGHsaWtlV20xlXQu7apvZVdvClqoGnlu1i/qWdiYNH8i108dyxdmju6Yl\nFq2r4DO/XMKYwREe++S7GJH/zhvKsm3VfGTuy/z7hyZzy3smHLD+O/+0ksde3cL8L1zI5BF5B9zW\n3dlR08zK7TWs2lHLqu01lNc0k5VpZGUYWZkZ8ekQIysjg8bWdrbsaWRHTROJkTEoO4twZgZ7euhf\nFc7MoLUjymmj8rjzslN514kHnp7stLu+hYKcUJ9/LXZO81Q3tfLcF/ed5tlR3cQtvyxl1Y5avvbB\nyXzmohN7/L8QjTpLtu7l7mdWs6KshiljCvjmv5zK2WMLk6p1waqd3PHkmzS0tPP1mZOZff64g37T\ncHfaox4L/yiHfFbSUZ3e6bZ9JlDl7vnx5Txigf/f7v5EXwUdSyP9Q9HWEaWiroXmtg7GDo7sN//a\nyd15aeNufrBwPW9sraa4IIfbLj6J+W+W89LG3fzwqil85OziQ67jkX++zTfnreLrMyczoSiXL/52\nGXnZIR68sYTTi/P32z4adX6wcD33LtrIOeMK+en10xg6cABtHVE27Kpn5fYaVmyv5s2yGtaU19Ha\nEWVIbpgPnj6CS08fwbknDtnntZa+XcX9L2xm4ZpdhDIz+OjU0VwzfQz1ze28tachNlLcHfu5paqB\n5rZYsGZmGBOHDeT04nxOH5XHGaPzOXVk/kEdU3nopbe465nVfO9jZ/G+ycPIywmR2cMvYnlNE69s\n2hO7bN5D2d7YX2MDsjIYNySXE4ZEGD80lxOG5DJuSISxQyJU1LWwdMte3thazdKteymvaQZiAReN\n//J2NyQ3zEUnF3Hdu8YydWxhj+Hz6uY9fOqRUgpzQ/zqk+cydkjsDe6WR0t57a0qXp5zMQMHHPiv\npurGVmZ873nGDcll1lmjiLrjDlF3ovGfNU1trN4Re2PtbD2SYTChaCBjB0focKetI0pbh9PeEaU9\n6rS2R8kJZ3btk3FDchkb/1kYCWFmtLR3UFEbe6Mrr2lmV20zO2uamTwyjyvOLu5x//eHnqZ5lmyp\n4tO/XEpzWwc/unoK7zul77+Uo1HnyTe2852/rKWiroUrzi7mazNP3u/UbXdnd30r26ub+NWrW/j9\nkjJOG5XHPR+fwsThqT1duj9DP4vYlM37gO3EDuRe6+6rErYZ6e7l8etXAF9393PNLAw8Czzt7vck\nU/jxHvoHy935x/pKfrhwPcvLagD4zkfP5Kpzxhz24972mzd49s1yHDizOJ8HPlHS57TEvOU7+Orv\nlzM4N8yI/GxW76ilJT7SHTQgi9OL8zlzdD7vnTyMc8YN7vOXeVNlPT9/6S2eWFK2z4g5nJXBCYMj\nXYE6viiX00blM3nEoMM+2NV5UHftzrqudYMGZJGXEyI/ftlZ28xbuxsAyM8J8a7xgzlvwhDOmzCE\nScMGJT2nGxQ7AAAFk0lEQVRaK69p4o2t1SzfVk1mhjE8L5vheQMYlpfN8LxsigYO6HNKqtPybdXc\n+IvXGZCVwWOffBcOfOCHL/CF903kS5dMSuoxfle6jTl/WEEP7z1A7M1p0oiBnD4qn9NG5XFacT6n\njMhLyYkK/SVxmmd3fSvf+ONKRhZk8+AnSg46iBta2vnJ8xt54MW3yDTjqpLRtHZEKdvbxPa9TWyv\nbur6fcgwuHXGSXz+fROT/jc+kvot9OMP9iHgHiATeMjdv2VmdwGl7j7PzL4NzALagSrgs+6+1syu\nB34BJJ5UO9vdl/X2XEEL/U7uzvPrK4lGPamRSTLqW9q5/sHXOLEol/++4oykw3RFWTXfeGol2aFM\nzizO54z4GQgnDI4c9J+unXbXt/DihkqGD8pm3NBcRuRlH/JjJaOqoZVFayuoaWrrutQmXC+IhDj3\nxFjInzIi74jWcjDW7azj+p+/RkfUmTR8ICvKanj56xcf1MHZuuY2olGwDMgwI8NiP80gKyPjiI26\nU6VzmmdrVSNNbR1ccNIQ5l47dZ/jWQdrW1Uj//PsWuavLGdIbpjighyKC3NiPwtyKC6McPLwQV1/\nkR0L+jX0j6aghr5Ip7d3N3Ddg6+xvbqJmy8cz398uPczliTmzbIarnngVa6cNppvfPiUfjt7rCPq\nx82bpEJf5Di2o7qJX7z8Fp+dcVJXJ1g5sPaO6BE/D/9YlmzoH/ufYBEJoFEFORrhH6QgB/7B0F4S\nEQkQhb6ISIAo9EVEAkShLyISIAp9EZEAUeiLiASIQl9EJEAU+iIiAXLMfSLXzCqBw+mtPBRIzy8M\nPXTaJ/vTPtmf9sn+jqd9coK79/lNSMdc6B8uMytN5qPIQaJ9sj/tk/1pn+wvHfeJpndERAJEoS8i\nEiDpGPr3p7qAY5D2yf60T/anfbK/tNsnaTenLyIivUvHkb6IiPQibULfzGaa2Toz22hmc1JdT6qY\n2UNmVmFmKxPWDTazhWa2If6zMJU1Hk1mNsbMFpnZajNbZWZfiK8P7D4BMLNsM3vdzJbH98t/xdeP\nN7PX4r9Hv41/z3WgmFmmmb1hZs/El9Nqn6RF6JtZJjAXuBQ4FbjGzIL6DRQPAzO7rZsD/M3dJwJ/\niy8HRTvwFXc/FTgX+Fz8/0aQ9wlAC3Cxu58FTAFmmtm5wP8CP3T3k4C9wCdTWGOqfAFYk7CcVvsk\nLUIfmA5sdPfN7t4KPA5cnuKaUsLdXyD25fSJLgceiV9/BPjIUS0qhdy93N2Xxq/XEftlLibA+wTA\nY+rji6H4xYGLgSfi6wO3X8xsNPBh4MH4spFm+yRdQr8Y2JawXBZfJzHD3b08fn0nMDyVxaSKmY0D\nzgZeQ/ukcxpjGVABLAQ2AdXu3h7fJIi/R/cAXwOi8eUhpNk+SZfQlyR57HStwJ2yZWYDgT8AX3T3\n2sTbgrpP3L3D3acAo4n9tTw5xSWllJldBlS4+5JU13IkpcsXo28HxiQsj46vk5hdZjbS3cvNbCSx\nkV1gmFmIWOD/yt2fjK8O9D5J5O7VZrYIOA8oMLOs+Mg2aL9HFwCzzOxDQDaQB/yINNsn6TLSXwxM\njB9lDwNXA/NSXNOxZB5wY/z6jcCfUljLURWfk/05sMbdf5BwU2D3CYCZFZlZQfx6DnAJseMdi4Ar\n45sFar+4+x3uPtrdxxHLkL+7+3Wk2T5Jmw9nxd+d7wEygYfc/VspLiklzOw3wAxi3QF3Ad8EngJ+\nB4wl1sH0KnfvfrA3LZnZu4EXgTd5Z57234nN6wdynwCY2ZnEDkpmEhv8/c7d7zKzE4mdCDEYeAO4\n3t1bUldpapjZDOB2d78s3fZJ2oS+iIj0LV2md0REJAkKfRGRAFHoi4gEiEJfRCRAFPoiIgGi0BcR\nCRCFvohIgCj0RUQC5P8DbSFTf2u3r+kAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x112782f28>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "training_losses = train_network(num_epochs = 5)\n",
    "plt.plot(training_losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Sources (and additional links for reading): \n",
    "\n",
    "1. https://r2rt.com/recurrent-neural-networks-in-tensorflow-i.html\n",
    "2. http://karpathy.github.io/2015/05/21/rnn-effectiveness/\n",
    "3. http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}