RecurrentNeuralNetwork.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Recurrent Neural Networks\n",
    "\n",
    "First, let's start with a toy example. The dataset is comprised of two sentences. Let's train a simple recurrent neural network to predict words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wed May 10 20:20:24 2017       \r\n",
      "+-----------------------------------------------------------------------------+\r\n",
      "| NVIDIA-SMI 367.57                 Driver Version: 367.57                    |\r\n",
      "|-------------------------------+----------------------+----------------------+\r\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\r\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\r\n",
      "|===============================+======================+======================|\r\n",
      "|   0  Tesla K80           On   | 0000:00:1E.0     Off |                    0 |\r\n",
      "| N/A   25C    P8    28W / 149W |      0MiB / 11439MiB |      0%      Default |\r\n",
      "+-------------------------------+----------------------+----------------------+\r\n",
      "                                                                               \r\n",
      "+-----------------------------------------------------------------------------+\r\n",
      "| Processes:                                                       GPU Memory |\r\n",
      "|  GPU       PID  Type  Process name                               Usage      |\r\n",
      "|=============================================================================|\r\n",
      "|  No running processes found                                                 |\r\n",
      "+-----------------------------------------------------------------------------+\r\n"
     ]
    }
   ],
   "source": [
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "import inspect\n",
    "import time\n",
    "\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow.python.framework import ops\n",
    "from tensorflow.python.framework import dtypes\n",
    "#import reader\n",
    "import collections\n",
    "import os\n",
    "import re\n",
    "import json\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "one-hot encoded inputs\n",
      "[[[ 0.  1.  0.  0.  0.  0.  0.  0.  0.  0.]\n",
      "  [ 0.  0.  1.  0.  0.  0.  0.  0.  0.  0.]\n",
      "  [ 0.  0.  0.  1.  0.  0.  0.  0.  0.  0.]\n",
      "  [ 0.  0.  0.  0.  1.  0.  0.  0.  0.  0.]\n",
      "  [ 0.  0.  0.  0.  0.  1.  0.  0.  0.  0.]\n",
      "  [ 1.  0.  0.  0.  0.  0.  0.  0.  0.  0.]]\n",
      "\n",
      " [[ 0.  1.  0.  0.  0.  0.  0.  0.  0.  0.]\n",
      "  [ 0.  0.  1.  0.  0.  0.  0.  0.  0.  0.]\n",
      "  [ 0.  0.  0.  1.  0.  0.  0.  0.  0.  0.]\n",
      "  [ 0.  0.  0.  0.  1.  0.  0.  0.  0.  0.]\n",
      "  [ 0.  0.  0.  0.  0.  0.  1.  0.  0.  0.]\n",
      "  [ 1.  0.  0.  0.  0.  0.  0.  0.  0.  0.]]]\n",
      "shape of the input\n",
      "(2, 6, 10)\n",
      "reshaped input for training\n",
      "[[array([[ 0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],\n",
      "       [ 0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]]), array([[ 0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],\n",
      "       [ 0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]]), array([[ 0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.],\n",
      "       [ 0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.]]), array([[ 0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.],\n",
      "       [ 0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.]]), array([[ 0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.],\n",
      "       [ 0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.]]), array([[ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],\n",
      "       [ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]])]]\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "a=np.array([[1,2,3,4,5,0],[1,2,3,4,6,0]]) \n",
    "init_op = tf.group(tf.global_variables_initializer(),\n",
    "                   tf.local_variables_initializer())    \n",
    "\n",
    "\n",
    "with tf.Session() as sess:\n",
    "        \n",
    "        sess.run(init_op)\n",
    "        coord = tf.train.Coordinator()\n",
    "        threads = tf.train.start_queue_runners(coord=coord)\n",
    "        result=tf.nn.embedding_lookup(np.identity(10), a).eval()\n",
    "        example_input=sess.run([tf.unstack(result,6,1)])\n",
    "        coord.request_stop()\n",
    "        coord.join(threads)\n",
    "        sess.close()\n",
    "print('one-hot encoded inputs')\n",
    "print(result)\n",
    "print('shape of the input')\n",
    "print(result.shape)\n",
    "print('reshaped input for training')\n",
    "print(example_input)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a single layer RNN with LSTMs and train it with a toy dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iteration:  0  loss:  2.11139\n",
      "iteration:  25  loss:  0.467161\n",
      "iteration:  50  loss:  0.143555\n",
      "iteration:  75  loss:  0.120925\n",
      "iteration:  100  loss:  0.118355\n",
      "iteration:  125  loss:  0.117469\n",
      "iteration:  150  loss:  0.116982\n",
      "iteration:  175  loss:  0.116668\n",
      "iteration:  200  loss:  0.11645\n",
      "iteration:  225  loss:  0.116292\n",
      "iteration:  250  loss:  0.116173\n",
      "iteration:  275  loss:  0.116081\n",
      "[[ 0.  0.  1.  0.  0.  0.  0.  0.]\n",
      " [ 0.  0.  1.  0.  0.  0.  0.  0.]\n",
      " [ 0.  0.  0.  0.  0.  0.  0.  1.]\n",
      " [ 0.  0.  0.  1.  0.  0.  0.  0.]\n",
      " [ 0.  0.  0.  0.  1.  0.  0.  0.]\n",
      " [ 0.  0.  0.  0.  1.  0.  0.  0.]\n",
      " [ 0.  0.  0.  0.  0.  1.  0.  0.]\n",
      " [ 0.  0.  0.  0.  0.  0.  1.  0.]\n",
      " [ 1.  0.  0.  0.  0.  0.  0.  0.]\n",
      " [ 1.  0.  0.  0.  0.  0.  0.  0.]\n",
      " [ 1.  0.  0.  0.  0.  0.  0.  0.]\n",
      " [ 1.  0.  0.  0.  0.  0.  0.  0.]]\n",
      "[2 2 7 7 4 4 5 6 0 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "plot_loss=[]\n",
    "num_hidden=24\n",
    "num_steps=6\n",
    "dict_length=8\n",
    "batch_size=2\n",
    "tf.reset_default_graph()\n",
    "\n",
    "## Make Variables\n",
    "variables_dict = {\n",
    "    \"weights1\":tf.Variable(tf.truncated_normal([num_hidden,dict_length],stddev=1.0,dtype=tf.float32),name=\"weights1\"),\n",
    "    \"biases1\": tf.Variable(tf.truncated_normal([dict_length],stddev=1.0,dtype=tf.float32), name=\"biases1\")}\n",
    "\n",
    "\n",
    "# Create input data\n",
    "small_dict=['EOS','i','will','walk','the','dog','cat','run']\n",
    "X=np.array([[1,2,7,4,5,0],[1,2,3,4,6,0]],dtype=np.int32)  \n",
    "X_one_hot=tf.nn.embedding_lookup(np.identity(dict_length), X) #[batch,num_steps,dictionary_length][2,6,7]\n",
    "y=np.zeros((batch_size,num_steps),dtype=np.int32)\n",
    "y[:,:-1]=X[:,1:]\n",
    "y_one_hot=tf.unstack(tf.nn.embedding_lookup(np.identity(dict_length), y),num_steps,1) #[batch,num_steps,dictionary_length][2,6,7]\n",
    "\n",
    "y_target_reshape=tf.reshape(y_one_hot,[batch_size*num_steps,dict_length])\n",
    "\n",
    "#Create our LSTM\n",
    "cell = tf.contrib.rnn.LSTMCell(num_units=num_hidden, state_is_tuple=True)\n",
    "\n",
    "outputs, last_states = tf.contrib.rnn.static_rnn(\n",
    "    cell=cell,\n",
    "    dtype=tf.float32,\n",
    "    inputs=tf.unstack(tf.to_float(X_one_hot),num_steps,1))\n",
    "\n",
    "output_reshape=tf.reshape(outputs, [batch_size*num_steps,num_hidden]) #[12==batch_size*num_steps,num_hidden==12]\n",
    "pred=tf.matmul(output_reshape, variables_dict[\"weights1\"]) +variables_dict[\"biases1\"]\n",
    "# Define loss and optimizer\n",
    "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y_target_reshape))\n",
    "optimizer = tf.train.AdamOptimizer(0.01).minimize(cost)\n",
    "\n",
    "\n",
    "init_op = tf.group(tf.global_variables_initializer(),\n",
    "                   tf.local_variables_initializer())    \n",
    "\n",
    "\n",
    "with tf.Session() as sess:\n",
    "        \n",
    "        sess.run(init_op)\n",
    "        coord = tf.train.Coordinator()\n",
    "        threads = tf.train.start_queue_runners(coord=coord)        \n",
    "        for i in range(300):\n",
    "            loss,_,y_target,y_pred,output=sess.run([cost,optimizer,y_target_reshape,pred,outputs])\n",
    "            plot_loss.append([loss])\n",
    "\n",
    "            if i% 25 ==0:\n",
    "                print(\"iteration: \",i,\" loss: \",loss)\n",
    "                \n",
    "        print(y_target)\n",
    "        print(np.argmax(y_pred,1))          \n",
    "        coord.request_stop()\n",
    "        coord.join(threads)\n",
    "        sess.close()    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input Sentence\n",
      "['i', 'will', 'run', 'the', 'dog', 'EOS']\n",
      "Predicted words\n",
      "['will', 'run', 'the', 'dog', 'EOS', 'EOS']\n"
     ]
    }
   ],
   "source": [
    "#Lets look at one input data point at each step and its prediction\n",
    "print(\"Input Sentence\")\n",
    "print([small_dict[ind] for ind in X[0,:]])\n",
    "print(\"Predicted words\")\n",
    "print([small_dict[ind] for ind in np.argmax(y_pred[0::2],1)])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are going to increase the depth of our RNN. Let's train an RNN with 2 and 4 layers. What parameter do you need to set to change the number of layers in your RNN? For a hint look [here](#answer1 \"num_layers=2 or num_layer=4\").\n",
    "\n",
    "Where do you enter the dropout values for the RNN?  For a hint look [here](#answer2 \"lstm_cell = tf.contrib.rnn.DropoutWrapper(lstm_cell,input_keep_prob=input_keep_prob,output_keep_prob=output_keep_prob)\").                         "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iteration:  0  loss:  2.66218\n",
      "iteration:  25  loss:  0.877686\n",
      "iteration:  50  loss:  0.347055\n",
      "iteration:  75  loss:  0.137709\n",
      "iteration:  100  loss:  0.120187\n",
      "iteration:  125  loss:  0.118272\n",
      "iteration:  150  loss:  0.117466\n",
      "iteration:  175  loss:  0.116932\n",
      "iteration:  200  loss:  0.116567\n",
      "iteration:  225  loss:  0.116353\n",
      "iteration:  250  loss:  0.116217\n",
      "iteration:  275  loss:  0.116118\n",
      "[[ 0.  0.  1.  0.  0.  0.  0.  0.]\n",
      " [ 0.  0.  1.  0.  0.  0.  0.  0.]\n",
      " [ 0.  0.  0.  0.  0.  0.  0.  1.]\n",
      " [ 0.  0.  0.  1.  0.  0.  0.  0.]\n",
      " [ 0.  0.  0.  0.  1.  0.  0.  0.]\n",
      " [ 0.  0.  0.  0.  1.  0.  0.  0.]\n",
      " [ 0.  0.  0.  0.  0.  1.  0.  0.]\n",
      " [ 0.  0.  0.  0.  0.  0.  1.  0.]\n",
      " [ 1.  0.  0.  0.  0.  0.  0.  0.]\n",
      " [ 1.  0.  0.  0.  0.  0.  0.  0.]\n",
      " [ 1.  0.  0.  0.  0.  0.  0.  0.]\n",
      " [ 1.  0.  0.  0.  0.  0.  0.  0.]]\n",
      "[2 2 7 7 4 4 5 6 0 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "#Now let's try multiple layers \n",
    "plot_loss2=[]\n",
    "num_hidden=24\n",
    "num_steps=6\n",
    "dict_length=8\n",
    "batch_size=2\n",
    "num_layers=2\n",
    "tf.reset_default_graph()\n",
    "\n",
    "## Make Variables\n",
    "variables_dict = {\n",
    "    \"weights1\":tf.Variable(tf.truncated_normal([num_hidden,dict_length],stddev=1.0,dtype=tf.float32),name=\"weights1\"),\n",
    "    \"biases1\": tf.Variable(tf.truncated_normal([dict_length],stddev=1.0,dtype=tf.float32), name=\"biases1\")}\n",
    "\n",
    "\n",
    "# Create input data\n",
    "small_dict=['EOS','i','will','walk','the','dog','cat','run']\n",
    "X=np.array([[1,2,7,4,5,0],[1,2,3,4,6,0]],dtype=np.int32)  \n",
    "X_one_hot=tf.nn.embedding_lookup(np.identity(dict_length), X) #[batch,num_steps,dictionary_length][2,6,7]\n",
    "y=np.zeros((batch_size,num_steps),dtype=np.int32)\n",
    "y[:,:-1]=X[:,1:]\n",
    "y_one_hot=tf.unstack(tf.nn.embedding_lookup(np.identity(dict_length), y),num_steps,1) #[batch,num_steps,dictionary_length][2,6,7]\n",
    "y_target_reshape=tf.reshape(y_one_hot,[batch_size*num_steps,dict_length])\n",
    "\n",
    "input_keep_prob=1.0#input_keep_prob,\n",
    "output_keep_prob=1.0#output_keep_pro\n",
    "\n",
    "##################### Create a multilayer RNN ####################\n",
    "layer_cell=[]\n",
    "for _ in range(num_layers):\n",
    "    lstm_cell = tf.contrib.rnn.LSTMCell(num_units=num_hidden, state_is_tuple=True)\n",
    "    lstm_cell = tf.contrib.rnn.DropoutWrapper(lstm_cell,\n",
    "                                          input_keep_prob=input_keep_prob,\n",
    "                                          output_keep_prob=output_keep_prob)\n",
    "    layer_cell.append(lstm_cell)\n",
    "\n",
    "cell = tf.contrib.rnn.MultiRNNCell(layer_cell, state_is_tuple=True)\n",
    "outputs, last_states = tf.contrib.rnn.static_rnn(\n",
    "    cell=cell,\n",
    "    dtype=tf.float32,\n",
    "    inputs=tf.unstack(tf.to_float(X_one_hot),num_steps,1))\n",
    "\n",
    "output_reshape=tf.reshape(outputs, [batch_size*num_steps,num_hidden]) #[12==batch_size*num_steps,num_hidden==12]\n",
    "pred=tf.matmul(output_reshape, variables_dict[\"weights1\"]) +variables_dict[\"biases1\"]\n",
    "\n",
    "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y_target_reshape))\n",
    "optimizer = tf.train.AdamOptimizer(0.01).minimize(cost)\n",
    "\n",
    "\n",
    "init_op = tf.group(tf.global_variables_initializer(),\n",
    "                   tf.local_variables_initializer())    \n",
    "\n",
    "\n",
    "with tf.Session() as sess:\n",
    "        \n",
    "        sess.run(init_op)\n",
    "        coord = tf.train.Coordinator()\n",
    "        threads = tf.train.start_queue_runners(coord=coord)\n",
    "        for i in range(300):\n",
    "            loss,_,y_target,y_pred,output=sess.run([cost,optimizer,y_target_reshape,pred,outputs])\n",
    "            plot_loss2.append([loss])\n",
    "            \n",
    "            if i% 25 ==0:\n",
    "                print(\"iteration: \",i,\" loss: \",loss)\n",
    "                \n",
    "        print(y_target)\n",
    "        print(np.argmax(y_pred,1))         \n",
    "        coord.request_stop()\n",
    "        coord.join(threads)\n",
    "        sess.close()    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Compare the loss from our single and multi-layer RNN. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VNX5x/HPkw0UEdkEBBRUpKBAwIBORZ0qspWi/rTu\nLKLl54JVW7XYulBsqz+ttWqtSt0VoW61WFsVkXEjKBHZhAqoqKAIBAVR2ZLn98e9iUNIyCRkMpPk\n+3697isz59zluVzIwz3n3HPN3REREalMRqoDEBGRukEJQ0REEqKEISIiCVHCEBGRhChhiIhIQpQw\nREQkIUoYIrvJzH5tZvelOo7KmNmBZrappteVhsP0HIbUNjNbAZzv7i/X8nFHA/cD34VFa4EYcKO7\nL63NWCpjZvsDi+OKmgDfAiX/YIe4++u1Hpg0aLrDkIYm3933ApoBAwiSxztmdlh1dmZmWTUZXAl3\n/8Td9ypZwuJecWU7JQszy0xGLCIllDAkrZjZz8xsuZmtN7NpZrZfWG5mdpuZrTGzjWa2sOSXvJkN\nNbPFZva1ma0ysysqO467F7n7B+5+EfAqMCHcV9TMVpaJaYWZDQg/TzCzp8zsMTPbCIwOyx4L6zuZ\nmZvZKDP7xMzWmdlv4va1h5k9bGZfmtkSM7uq7PGq8Gf1mJndZWYvmNk3wNFmNtzM5oV/Rp+Y2bVx\n6x9sZh73/Q0z+62ZzQr/7F4wsxZVXTesPzfufH9tZivNLFqd85L0pYQhacPMjgNuBE4D2gEfA1PD\n6oHAMcAhBHcHpwGFYd39wP+6e1PgMOCVKh76GeDoKqx/IvAUsA8wuYJ1+gNdgeOB68ysW1h+PdAJ\nOBA4ATinirGWdRbwW6ApkA9sAs4OY/sJcKmZDatk+1FAG4Jmr19UdV0z6wHcAZwBtAdaA22rfUaS\ntpQwJJ2cDTzg7nPdfQtwNRAxs07ANoJfij8g6Htb4u6fh9ttA7qb2d7u/qW7z63icT8DWlS61vfy\n3f1Zdy929+8qWOe37v6du88H5gO9wvLTgD+Eca4k+EW7O/7h7vlhLFvc/RV3fy/8Pp8g4R67i+3v\nd/dl7v4t8CSQW411fwo86+6zwut2zW6ek6QpJQxJJ/sR3FUA4O6bCO4i2rv7K8BfgLuANWY2ycz2\nDlc9BRgKfGxmr5pZpIrHbQ+sr8L6nyawzuq4z98CJf0Q+5XZPpF9JRyLmUXMLGZma81sA3A+0Koa\ncVZl3R3Oyd2/Ab5MIHapY5QwJJ18BhxQ8sXMmgAtgVUA7n6Hux8OdCdomroyLJ/j7icC+wLPAk9U\n8bgnAyWdyN8Ae8bFkEnQxBJvd4YWfg50iPvecTf2VV4sU4GngY7u3gy4D7DdPEZldjin8Lo1T/Ix\nJQWUMCRVss2scdySBUwBzjWzXDNrBPwBeMvdV5hZXzM7wsyyCX6pbwaKzSzHzM42s2buvg3YCBRX\ndnAzyzSzzmZ2JxAl6AcAWAo0NrMfh8e6BmhUg+f9BHC1mTU3s/bAuBrcNwTNduvdfbOZHUnQr5Bs\nTwInmdmRZpYDTKyFY0oKKGFIqvybYEhryTIhfC7jWoL/IX8OHMT3v/D2Bv5G0NTxMUFT1S1h3Qhg\nRThq6QKCvpCKRMIH0jYSPIOxN9DX3RcCuPsG4CKC/5mvIkhO1RrFVIGJ4f4+Al4m6DzfUoP7vxC4\n0cy+Bn5N1e+2qszdFwCXEySOzwiuTSE1e16SBvTgnkgKmdmFwBnuvquO6Tol7Fv6CjjA3Xe3j0bS\niO4wRGqRmbUzs6PMLMPMugK/BP6R6rh2V/j8x55mthdwKzBXyaL+UcIQqV05wL3A1wTPi/wT+GtK\nI6oZJxM0R60keM7kzJRGI0mhJikREUmI7jBERCQhSZk4LVVatWrlnTp1SnUYIiJ1xjvvvLPO3cs+\na1SuepUwOnXqREFBQarDEBGpM8zs48rXCqhJSkREEqKEISIiCVHCEBGRhNSrPgwRqb5t27axcuVK\nNm/enOpQJAkaN25Mhw4dyM7OrvY+lDBEBICVK1fStGlTOnXqhFmyJ7iV2uTuFBYWsnLlSjp37lzt\n/ahJSkQA2Lx5My1btlSyqIfMjJYtW+723aMSBkB+Ptx4Y/BTpAFTsqi/auLaqkkqPx+OPx62boWc\nHJgxAyJVfWGbiEj9pzuMWIyTvpvCH4suC5JGLJbqiEQarDFjxrDvvvty2GGH7XK9vfba1Ztkk2fF\nihXlxjZ79myOOOIIcnNz6datGxMmTODBBx8kNzeX3NxccnJy6NGjB7m5uYwfP56HHnoIM+Pll18u\n3cezzz6LmfHUU0/V5ilVie4wolEWWhua8nVwhxGNpjoikQZr9OjRjBs3jpEjR6Y6FAC2b99OVlbl\nvyZHjRrFE088Qa9evSgqKuL999+ne/funHvuuUAwC8XMmTNp1Sp4vfpDDz1Ejx49mDp1KgMGDABg\nypQp9OrVK3knUwN0hxGJ0PIH+7KuS0TNUSJVVcP9f8cccwwtWrSo1rbPPfccRxxxBL1792bAgAF8\n8cUXFBcX06VLF9auXQtAcXExBx98MGvXrmXt2rWccsop9O3bl759+/Lmm28CMGHCBEaMGMFRRx3F\niBEjEjr2mjVraNeuHQCZmZl079690m2OPvpo3n77bbZt28amTZtYvnw5ubm51Tr32qI7DKBVp71Y\nt24viByU6lBE6o406//r378/s2fPxsy47777uPnmm7n11ls555xzmDx5Mpdddhkvv/wyvXr1onXr\n1px11llcfvnl9O/fn08++YRBgwaxZMkSABYvXswbb7zBHnvskdCxL7/8crp27Uo0GmXw4MGMGjWK\nxo0b73IbM2PAgAG8+OKLbNiwgeHDh/PRRx/t9p9DMukOA2jZEtatS3UUInVMLBYki6KitOj/W7ly\nJYMGDaJHjx7ccsstvPfee0DQL/LII48A8MADD5Q2E7388suMGzeO3Nxchg8fzsaNG9m0aRMAw4cP\nTzhZAFx33XUUFBQwcOBAHn/8cQYPHpzQdmeccQZTp05l6tSpnHlm+r9zSgkDOOQQ6Ngx1VGI1DHR\naHBnkZmZ1P6/Tz/9tLTz+J577qlwvUsuuYRx48axcOFC7r333tJnDjp27EibNm145ZVXePvttxky\nZAgQNE/Nnj2befPmMW/ePFatWlXamd6kSZMqx3nQQQdx4YUXMmPGDObPn09hYWGl2/Tr14+FCxey\nbt06DjnkkCofs7apSQq49tpgEZEqiIT9frFYkCyS1BzVsWNH5s2bV+l6GzZsoH379gA8/PDDO9Sd\nf/75nHPOOYwYMYLMzEwABg4cyJ133smVV14JwLx586rdh/D8888zdOhQzIxly5aRmZnJPvvsk9C2\nN910U6XNV+lCdxgiUn2RCFx9dY0lizPPPJNIJML7779Phw4duP/++8td79tvv6VDhw6ly5/+9Ccm\nTJjAT3/6Uw4//PDS0Uglhg8fzqZNm0qbowDuuOMOCgoK6NmzJ927d9/l3Uu8kthKlieffJJHH32U\nrl27kpuby4gRI5g8eXJpYqrMkCFD+NGPfpTQuinn7klZgI7ATGAx8B5waTnrGHAHsBxYAPSJqxsF\nLAuXUYkc8/DDD/fqeO0197593f/732ptLlIvLF68ONUhJM2cOXO8f//+qQ4j5cq7xkCBJ/h7PZlN\nUtuBX7r7XDNrCrxjZtPdfXHcOkOALuFyBHA3cISZtQCuB/IAD7ed5u5fJiPQbdtgzhz4/HPo2jUZ\nRxCRVLnpppu4++67mTx5cqpDqfOS1iTl7p+7+9zw89fAEqB9mdVOBB4JE91sYB8zawcMAqa7+/ow\nSUwHEht2UA0ld68J9FGJSB0zfvx4Pv74Y/r375/qUOq8WunDMLNOQG/grTJV7YFP476vDMsqKi9v\n32PNrMDMCkoezqmqli2DnxpaKyJSsaQnDDPbC3gauMzdN9b0/t19krvnuXte69atq7WPkoRR+ExM\nM9aKiFQgqQnDzLIJksVkd3+mnFVWEXSOl+gQllVUnhSN382nrxWw9/RngidXlTRERHaStIRhweTr\n9wNL3P1PFaw2DRhpgSOBDe7+OfAiMNDMmptZc2BgWJYcsRhvZxzJOL8zLZ5YFRFJR8m8wzgKGAEc\nZ2bzwmWomV1gZheE6/wb+JBgWO3fgIsA3H09cAMwJ1wmhmXJUUtPrIrIrpkZ55xzTun37du307p1\na4YNG1bptiVPaa9YsYLHH3+8tLygoICf//zn5W4TjUYpKCjYzairp1OnTqwr03H6xRdfMGzYMHr1\n6kX37t0ZOnQoCxcuLH3SvUWLFnTu3Jnc3FwGDBjAihUrMDOuueaa0n2sW7eO7Oxsxo0bV+MxJ21Y\nrbu/QfCcxa7WceDiCuoeAB5IQmg7i0T4xfDlrFr8FX+/d4NmrBVJkSZNmrBo0SK+++479thjD6ZP\nn1769HaiShLGWWedBUBeXh55eXnJCDdhiU6Tft1113HCCSdw6aWXArBgwQJ69OhR+qT76NGjGTZs\nGKeeeioQnGvnzp15/vnn+d3vfgfAk08+yaGHHpqU89CT3qEvMvfjnW+7K1mIpNjQoUN5/vnngeAd\nEfGT8k2YMIE//vGPpd8PO+wwVqxYscP248eP5/XXXyc3N5fbbruNWCyW0B1KiRUrVnD00UfTp08f\n+vTpw6xZswAYOXIkzz77bOl6Z599Nv/85z8pKiriyiuvpG/fvvTs2ZN7770XgFgsxtFHH83w4cMT\nmu4c4PPPP6dDhw6l33v27FnpNnvuuSfdunUrvVP6+9//zmmnnZbw+VaF5pIKacZakR2V1zI7bBhc\ncUX16hPtGjzjjDOYOHEiw4YNY8GCBYwZM4bXX389sY0JHtT74x//yL/+9a/wuAkeOLTvvvsyffp0\nGjduzLJlyzjzzDMpKCjgvPPO47bbbuOkk05iw4YNzJo1i4cffpj777+fZs2aMWfOHLZs2cJRRx3F\nwIEDAZg7dy6LFi2ic+fOCR374osv5vTTT+cvf/kLAwYM4Nxzz2W//fardLuSWW/btGlDZmYm++23\nH5999lmVzjsRShihVq1gw4bgqe/s7FRHI9Jw9ezZkxUrVjBlyhSGDh1a68fftm0b48aNY968eWRm\nZrJ06VIAjj32WC666CLWrl3L008/zSmnnEJWVhYvvfQSCxYsKH216oYNG1i2bBk5OTn069cv4WQB\nMGjQID788ENeeOEF/vOf/9C7d28WLVpEZY8MDB48mGuvvZY2bdpw+umnV//kK6GEESp5FmP9emjT\nJrWxiKSDyv5jvrv1uzJ8+HCuuOIKYrHYDtOEZ2VlUVxcXPq9ZArzRA0aNIgvvviCvLw87rvvvnLX\nue2222jTpg3z58+nuLh4h5lkR44cyWOPPcbUqVN58MEHgWA+vjvvvJNBgwbtsJ9YLFatadJbtGjB\nWWedxVlnncWwYcN47bXXOOWUU3a5TU5ODocffji33norixcvZtq0aVU+biKUMEKdOwfdF1u3pjoS\nERkzZgz77LMPPXr02KFJqVOnTqVNTXPnzi33DXVNmzbl66+/Lne/L75Y+ej8DRs20KFDBzIyMnj4\n4YcpKioqrRs9ejT9+vWjbdu2pf0SgwYN4u677+a4444jOzubpUuXVrmjvsQrr7zCkUceyZ577snX\nX3/NBx98wP7775/Qtr/85S859thjq/2K20QoYYSGDg0WEUm9Dh06lDsU9pRTTuGRRx7h0EMP5Ygj\njij3pUM9e/YkMzOTXr16MXr0aHr37r3LY/34xz8mO2yHjkQi/OEPfyg9zuDBg3e4S2jTpg3dunXj\npJNOKi07//zzWbFiBX369MHdad269Q6d47vSs2dPMjKCsUennXYa7dq1Y9y4caV3Uueffz59+/ZN\naF+HHnpo0kZHlbBgZGv9kJeX56kaUy1S1y1ZsoRu3bqlOoy09u2339KjRw/mzp1Ls2bNUh1OlZV3\njc3sHXdPaNyxhtWG1q6Fnj1hypRURyIi6ejll1+mW7duXHLJJXUyWdQENUmFmjSBhQvh44dnQqfG\neh5DRHYwYMAAPv7441SHkVK6wwjtOT+fPfiWwpfmagJCabDqUxO17Kgmrq0SRolYjJYUss5baAJC\naZAaN25MYWGhkkY95O4UFhbuMES4OtQkVSIapZWtp9BbaQJCaZA6dOjAypUrqe6LyCS9NW7ceIdp\nR6pDCaNEJMIPT15Nzhdb4JYZ6sOQBic7O7tKTyVLw6OEEeeup9sCbVMdhohIWlIfhoiIJEQJI849\n98CBB0LcTAAiIhJK5itaHzCzNWa2qIL6K+PexLfIzIrMrEVYt8LMFoZ1tfbo9tat8NFH8OWXtXVE\nEZG6I5l3GA8BgyuqdPdb3D3X3XOBq4FXy7yG9Udhfa29KqtVq+Cn3oshIrKzpCUMd38NSPQ93GcC\nKZ+UoyRhxM2mLCIioZT3YZjZngR3Ik/HFTvwkpm9Y2ZjK9l+rJkVmFnB7o4fL3knhu4wRER2lvKE\nAfwEeLNMc1R/d+8DDAEuNrNjKtrY3Se5e56751X2VqrKtG0bPK/XtOlu7UZEpF5Kh+cwzqBMc5S7\nrwp/rjGzfwD9gNeSHUj79jBzZrKPIiJSN6X0DsPMmgHHAv+MK2tiZk1LPgMDgXJHWiVFfj7ceKMm\nHxQRKSNpdxhmNgWIAq3MbCVwPZAN4O73hKudDLzk7t/EbdoG+IeZlcT3uLu/kKw4d5CfzzH9izjc\n9+S2xsfDDE0RIiJSImkJw93PTGCdhwiG38aXfQj0Sk5UlYjF+Kr4x3zEAd/PWKuEISICpEend/qI\nRmmZ8SWFaMZaEZGy0qHTO31EIrQ6tpBFixz+qeYoEZF4ShhltDykJYWLgEirVIciIpJWlDDKOPxw\nWLMG3CHodxcREVAfxk5+9jN45hklCxGRspQwREQkIUoYZbz+OrRrB2+9lepIRETSixJGGY0awerV\nsJvzGIqI1DtKGGXonRgiIuVTwiijZIpzvRNDRGRHShhl7L03ZGUWs+65WZqAUEQkjhJGGTY7n1N5\nioNfexCOP15JQ0QkpAf3yorFmMK14EWwNVMTEIqIhHSHUVY0Gkw8mJmpCQhFROIoYZQVifC/Az+i\nb9tP9T4MEZE4apIqT5s2fLodiLRLdSQiImkjaXcYZvaAma0xs3Jfr2pmUTPbYGbzwuW6uLrBZva+\nmS03s/HJirEiLVsGw2rda/vIIiLpK5lNUg8BgytZ53V3zw2XiQBmlgncBQwBugNnmln3JMa5k1at\nYPt22LixNo8qIpLekpYw3P01YH01Nu0HLHf3D919KzAVOLFGg6uEHt4TEdlZqju9I2Y238z+Y2aH\nhmXtgU/j1lkZlpXLzMaaWYGZFaytoQmgunaFn/40GCglIiKBVHZ6zwUOcPdNZjYUeBboUtWduPsk\nYBJAXl5ejfQ6HHkkPPFETexJRKT+SNkdhrtvdPdN4ed/A9lm1gpYBXSMW7VDWCYiIimUsoRhZm3N\ngvfamVm/MJZCYA7Qxcw6m1kOcAYwrTZj27gRWrSA22+vzaOKiKS3pDVJmdkUIAq0MrOVwPVANoC7\n3wOcClxoZtuB74Az3N2B7WY2DngRyAQecPf3khVnefbaCzZudNY+Owv6ZejhPRERkpgw3P3MSur/\nAvylgrp/A/9ORlyJyHgrnxZFB7Hu1UVw/OV64ltEhNSPkkpPsRj7soY13hq2bg0mIBQRaeCUMMoT\njbJfxmo+o70mIBQRCSlhlCcSYeglB3HcsUVqjhIRCWnywQpc9ufOQOdUhyEikjZ0h7ELxcXBIiIi\nShgVev55aNQIFixIdSQiIulBCaMCLVsGM9Z+9lmqIxERSQ9KGBVoF747SQlDRCSghFGBtm2Dn5//\n/TXIz09tMCIiaUAJowKN5ubTirV8NmMxHH+8koaINHhKGBWJxRhjD/FDf1NPe4uIoOcwKhaN8n+N\njw+ShZ72FhFRwqhQJAIzZvDVf/JpcsIPyY4cmeqIRERSSk1Su/DcugjNb/gF8xorWYiIKGHsQpfw\nhbFLbn5Ond4i0uAlLWGY2QNmtsbMFlVQf7aZLTCzhWY2y8x6xdWtCMvnmVlBsmKszEFrZ5PFNpY8\nrZFSIiLJvMN4CBi8i/qPgGPdvQdwAzCpTP2P3D3X3fOSFF+lst+YSVfeZ7730EgpEWnwkpYw3P01\nYP0u6me5+5fh19lAh2TFUm3RKEdlvsWbHEVRdmONlBKRBi1dRkmdB/wn7rsDL5mZA/e6e9m7j9oR\niXD2HU3pNvkltnYfwx4pCUJEJD2kPGGY2Y8IEkb/uOL+7r7KzPYFppvZf8M7lvK2HwuMBdh///1r\nPL5jen/NMVeMgre2wuT79EIlEWmwUjpKysx6AvcBJ7p7YUm5u68Kf64B/gH0q2gf7j7J3fPcPa91\n69Y1H2QsxuotzXmz6Aj1Y4hIg5ayhGFm+wPPACPcfWlceRMza1ryGRgIlDvSqlZEo4y3mziJZ/Fs\nPfEtIg1XMofVTgHyga5mttLMzjOzC8zsgnCV64CWwF/LDJ9tA7xhZvOBt4Hn3f2FZMVZqUiEY38V\nYR2tWXL/LDVHiUiDlbQ+DHc/s5L684Hzyyn/EOi18xapc8yYg+EP8NoTq+neOV9JQ0QaJD3pnYAD\nv8hnPz7jtWlf6gE+EWmwlDASYK/GyKOA+d5THd8i0mAllDDM7FIz29sC95vZXDMbmOzg0kY0yoRG\nN/JoxmhNdS4iDVaidxhj3H0jwYil5sAI4KakRZVuIhF6z/wTfX73P3oOQ0QarEQ7vS38ORR41N3f\nMzPb1Qb1zebeEZ5YFqFHY+id6mBERFIg0TuMd8zsJYKE8WL4nERx8sJKPxkZcN6YYp68fJY6vUWk\nQUo0YZwHjAf6uvu3QDZwbtKiSkM57+RzSPF/WfRqoUZKiUiDlGjCiADvu/tXZnYOcA2wIXlhpaFY\njK7+PkvpopFSItIgJZow7ga+DV9y9EvgA+CRpEWVjqJRumR9yIccqKnORaRBSjRhbHd3B04E/uLu\ndwFNkxdWGopE6HLFSWwjh08ee00jpUSkwUk0YXxtZlcTDKd93swyCPoxGpSTrziIpUth/5P6pDoU\nEZFal2jCOB3YQvA8xmqCt+PdkrSo0lTLltClC2RmpjoSEZHal1DCCJPEZKCZmQ0DNrt7w+rDCE36\n1Qc8ffYzGiUlIg1OolODnEYw1fhPgdOAt8zs1GQGlpby8/nrLd/wwOONNbRWRBqcRJ/0/g3BMxhr\nAMysNfAy8FSyAktLsRhdvAvziZuEUJ3fItJAJNqHkVGSLEKFVdi2/ohG6ZL1ER/Rme3Ze2horYg0\nKIn+0n/BzF40s9FmNhp4Hvh3ZRuZ2QNmtsbMyn3Fajj77R1mttzMFphZn7i6UWa2LFxGJRhnckUi\nHHzVyWwnm48feVV3FyLSoCTa6X0lMAnoGS6T3P1XCWz6EDB4F/VDgC7hMpbgAUHMrAVwPXAE0A+4\n3syaJxJrsnUZfDAAy/fW0FoRaVgSfkWruz8NPF2Vnbv7a2bWaRernAg8Ej4UONvM9jGzdkAUmO7u\n6wHMbDpB4plSleMnQ79+sGYNtGqV6khERGrXLhOGmX0NeHlVgLv73rt5/PbAp3HfV4ZlFZWXF+NY\ngrsT9t9//90Mp3KNGkHr5flwXyzow1CzlIg0ELtMGO6e9tN/uPskguYy8vLyyktuNSs/n/uOfZT1\n2/fmqsbH64VKItJgpHqk0yqgY9z3DmFZReWpF4vxyvaj+atfoFlrRaRBSXXCmAaMDEdLHQlscPfP\ngReBgWbWPOzsHhiWpV40SvfMpXxMJzZlN9fQWhFpMBLu9K4OM5tC0IHdysxWEox8ygZw93sIhuYO\nBZYD3xK+lMnd15vZDcCccFcTSzrAUy4S4dAbWsDV8N+7ZpAX6ZnqiEREakVSE4a7n1lJvQMXV1D3\nAPBAMuLaXd1P7gpXw3uZPclLdTAiIrUk1U1SddJBB0HjnGJWPR7TfFIi0mAk9Q6jvsqak89XGUNo\nNGMTvJ6jkVIi0iDoDqM6YjEabdsERUUaKSUiDYYSRnVEo7yVdRRD+A+rsjtppJSINAhKGNURieB3\n/oUXGMzb1z+v5igRaRCUMKopd0QPsrOKefuFQnV8i0iDoIRRTY3fzadX0bvMfnWr3r4nIg2CEkZ1\nxWIcxRu8RT+2bEEd3yJS7ylhVFc0ynHZb9CdxXyevb86vkWk3tNzGNUViTA89guGP3I/8KNURyMi\nknS6w9hdDz9M0aT71Y8hIvWeEsbuiMX46+YxtC1exdYtrn4MEanXlDB2RzTKftlrWUdr5mRF1I8h\nIvWaEsbuiEQ4ZtoVGMXEel2a6mhERJJKCWM3tdh7Oz1tEbE5TdSPISL1mhLG7orFiDKTN/mh+jFE\npF5L9hv3BgO3A5nAfe5+U5n62/h+TOqewL7uvk9YVwQsDOs+cffhyYy12qJRTs6+nuxt2/k2Zx9y\n1I8hIvVU0hKGmWUCdwEnACuBOWY2zd0Xl6zj7pfHrX8J0DtuF9+5e26y4qsxkQjHxn7LsY88ApyU\n6mhERJImmU1S/YDl7v6hu28FpgIn7mL9M4EpSYwnqbY+9DiLJs1SP4aI1FvJTBjtgU/jvq8My3Zi\nZgcAnYFX4oobm1mBmc02swr/625mY8P1CtauXVsTcVddLMaVW27gyOI32balWP0YIlIvpUun9xnA\nU+5eFFd2gLvnAWcBfzazg8rb0N0nuXueu+e1bt26NmLdWTTKD7Pm8A17MT87T89jiEi9lMyEsQro\nGPe9Q1hWnjMo0xzl7qvCnx8CMXbs30gvkQhHPXkZAG9e8KheqCQi9VIyE8YcoIuZdTazHIKkMK3s\nSmb2A6A5kB9X1tzMGoWfWwFHAYvLbptOOpx4OAe03cwbMzarD0NE6qWkJQx33w6MA14ElgBPuPt7\nZjbRzOKHyJ4BTHV3jyvrBhSY2XxgJnBT/OiqtJSfz1Frn+XNRc3w49TxLSL1T1Kfw3D3fwP/LlN2\nXZnvE8rZbhbQI5mx1bhYjEv8X5zFo/jWbVgspqYpEalX9D6MmhKNcmSjG2DrVsjJUce3iNQ7Shg1\nJRKBGTPd36EYAAASCUlEQVTIv/l1vtq6J0NSHY+ISA1TwqhhE6fl8klxe4bM7AszZqhZSkTqjXR5\nDqN+iMU4zmewmENZtaWVHuATkXpFCaMmRaMMypkJwEuZQ9SPISL1ihJGTYpE6PHK7bTb8yte3P9n\nqY5GRKRGKWHUMDMYuOU5Yh900PMYIlKvKGHUtFiM3/tvWMoh2Lat6scQkXpDCaOmRaO0b7SOvTO+\nCW43WrZMdUQiIjVCCaOmRSLw5z/ziI3i4u23w2WXqVlKROoFJYxkKCzko+IDuIf/5Yst+6hZSkTq\nBSWMZIhG+Z+cf1FMJs9ykpqlRKReUMJIhkiEw27/GQezjKeLT1azlIjUC0oYSWLrCznF/sFMoqzf\n0kTNUiJS5ylhJEs0yqk50zicd/jM2qtZSkTqPCWMZIlEyLtjJLOzj+EwX6hmKRGp85KaMMxssJm9\nb2bLzWx8OfWjzWytmc0Ll/Pj6kaZ2bJwGZXMOJOmsBCKi/mquCnfbMlSs5SI1GlJSxhmlgncBQwB\nugNnmln3clb9u7vnhst94bYtgOuBI4B+wPVm1jxZsSZNNMpHWV1oy2oe5yw1S4lInZbMO4x+wHJ3\n/9DdtwJTgRMT3HYQMN3d17v7l8B0YHCS4kyeSIROt19OZ1bwaPFZapYSkTotmQmjPfBp3PeVYVlZ\np5jZAjN7ysw6VnFbzGysmRWYWcHatWtrIu4aZesLGWGP8TrHsGJLOzVLiUidlepO7+eATu7ek+Au\n4uGq7sDdJ7l7nrvntW7dusYD3G3RKGfnPAnAZDVLiUgdlsyEsQroGPe9Q1hWyt0L3X1L+PU+4PBE\nt60zIhEOuOOXHGOv8Wjx2filapYSkbopmQljDtDFzDqbWQ5wBjAtfgUzaxf3dTiwJPz8IjDQzJqH\nnd0Dw7K6qbCQm+zXPMoI2Kopz0WkbspK1o7dfbuZjSP4RZ8JPODu75nZRKDA3acBPzez4cB2YD0w\nOtx2vZndQJB0ACa6+/pkxZp00SiRRjfAli2QkaFmKRGpk8zdUx1DjcnLy/OCgoJUh1G+SZNYePE9\n/G37udza+Ddkv/JiMBW6iEgKmdk77p6XyLqp7vRuOAoL+aS4A3dyCc9tPgEeeSTVEYmIVIkSRm2J\nRhmcOZ32rOQ+zoMHH1Tnt4jUKUoYtSUSIfO80YzhQV5gMJ9s0zMZIlK3KGHUppEjGdNoMgAPMlqd\n3yJSpyhh1KZIhE53/IKTbBpbi7M1VYiI1ClJG1YrFSgs5Gm7CPMi2GxB57dGS4lIHaA7jNoWjWJZ\nmQAs8u7q/BaROkMJo7ZFIjBmDI8ygh4sYu62Hur8FpE6QQkjFUaOZHjjl9iLr7ndf67ObxGpE5Qw\nUiESodntEzk342Gm+Ol8+vNb1CwlImlPCSNVCgu5glsxnBu2XKUnv0Uk7SlhpEo0yv5Zn3EB9/A0\n/8PGB57SXYaIpDUljFQJO7+vZyJLOYS9txXqLkNE0poSRiqNHEmLnE20ZD3FDqsmPQ+TJqU6KhGR\ncilhpFJ4lwFwDo8xoPhFNl/8SzVNiUhaSmrCMLPBZva+mS03s/Hl1P/CzBab2QIzm2FmB8TVFZnZ\nvHCZVnbbemPkSMjKYhQP81+68fvtv1LTlIikpaQlDDPLBO4ChgDdgTPNrHuZ1d4F8ty9J/AUcHNc\n3Xfunhsuw5MVZ8pFInDXXQzKnMEoHuImfsXsSQvUNCUiaSeZdxj9gOXu/qG7bwWmAifGr+DuM939\n2/DrbKBDEuNJX2PHws9+xm1cTkc+5afFU1lzwXVKGiKSVpKZMNoDn8Z9XxmWVeQ84D9x3xubWYGZ\nzTazk5IRYFoZOZLmWZt4mlPIZhsrfT+48EIlDRFJG2nR6W1m5wB5wC1xxQeE75k9C/izmR1UwbZj\nw8RSsHbt2lqINknCpqneGQt4n6704V0oLsYvHqdOcBFJC8lMGKuAjnHfO4RlOzCzAcBvgOHuvqWk\n3N1XhT8/BGJA7/IO4u6T3D3P3fNat25dc9GnwtixcPfdZGcU48DV/IGJ23+tTnARSQvJTBhzgC5m\n1tnMcoAzgB1GO5lZb+BegmSxJq68uZk1Cj+3Ao4CFicx1vQRJg0yMllNWyYwgdvu3VNNUyKScklL\nGO6+HRgHvAgsAZ5w9/fMbKKZlYx6ugXYC3iyzPDZbkCBmc0HZgI3uXvDSBgAY8diY3/GJMZyKk/y\nC7+VP12wNOjTUPOUiKSIuXuqY6gxeXl5XlBQkOowakZ+PhxzDNu2w9lM5klO41omMjFzIvzkJ3DV\nVXpTn4jsNjN7J+wvrlRadHpLOcJO8Oxs43HOZiz30oGVUFQEzz4LRx+tZioRqVVKGOls7Fh49VWy\nLjifezMvZix/A+ApTiFW1B8uuABOPlnNVCJSK5Qw0l0kEnSC//WvkJlJMcZNjOdHxDjNp7Lo2WXQ\nv78Sh4gknRJGXTF2LLz+OhknnchrHMs13MALDKYHiziueDqxZ78MEsexxwbJQx3kIlLD1OldF02a\nBBddxPqivbmbC3mQc7mNy/kJ/2IZB/M6R/NDZtHFPiDz6B9Cixbfb9u2bTDhoTrMRYSqdXorYdRV\n+flw880wbRpeXIxjZODcyi+4glsB2JNv6MV8uvI+N3I1bfmCNbRmmzWiTf8uZLVsVnPxKBGJ1ElK\nGA1JXOKgOHhC/D0OpYA83qU388hlOQezgJ60ZD3XMpHfcS0ATdlIc76kOV8yg+NpyXqe4WSmcwKN\n2Vy67MF3XMKd7MFm5tGL9+lKFtvJYjuZFJHFdo5nBtkZxazIO5XPG3cmy4rIsiIyrZhMK+YHTVeR\nacWs29KUr7Y1wQwyKMZwzKBj5ywy+uSy8a0lfPPZBgwnwzyox2mRs4kMczYXZbOtOPP77Q0Mp1HG\nNsygyDOI/yttFvwsWTdl2raF3r3h3Xdh9eoUBpJEOsfU2Y3/sFUlYeDu9WY5/PDDvcGaNcv9ggvc\nTzrJ/Zhj3DMy3GGnZR49/a9c4L/lWr+U23wkD/lP+Kd/S2N38Bv4je/Lam/KBs9mS+mmG2jqDn4F\nN5e3W/+Kvd3Br+T/dllf2fapqv+SZu7gV3GTZ7DdM9jumWzzTLZ5FltL68fzB89hs+ew2RvxnTfi\nO2/Mt6X7/zW/8z3ZVLo04Wtvwtel9dcw0ffmq9KlGV96M74srb+OCd6cQm9OobdgnbdgnbdkbWn9\nBK7z1nzhrfnC92W178tqb8PnpfUTucbbsWqnpeT63cBvfD9W7rSU1P+OX3t7Pt1pia/vwCc7LSX1\nv+dq78jHOy3x9fuzYqelpP4PjPcD+GinpaT+Rn7lnflgpyW+/kCW77RsZC938Ju4yg9i2U5LfP3B\nLN1pKan/P670Lry/01JSfzNX+CH8d6clvr4rS3ZaSupv4Zf+AxbvtMTXd+O9nZaN7OXeqFHwe6CK\ngAJP8HdsVpXTkaSnSGTH/13k5wdzUMX/L2j9enq98Qa9ihdUuJtr+D3X8PvS70VksIVG7MF3AFzF\nzZzLgxSRGd5jZFFEJnuxCYDzuY/jeGWn+j0JZrE/kyn0Yj7FZOBY6c+S/Z/C0xzM8grrf8JztGX1\nTvWN2QzAIF6kGRsAcL6/pWhEME3ZCUynCd/sVF+y/Y+YSQ5bS+tKfpbUH83rO5SX/Mxha3AZyGcr\nOTvtv6S+L3MYwwMV1vfmXc7hsQr334OFnMpTFdZ3YwnD+BdlZbEdgB/wX4by7wrrD2Epg3lhl/UD\neanC+oNZzgBe3mX9cbxSYf2BfEiUWIX1nVhBf97YZf0PmbVTfSZFAOzPJxzJ7F3W9+PtCus7sJI8\ndm7FKKnfj8/ow9xd1ucyr8L6tqymJzv/+4yvP4xF5ddv3QqxWFKbhdUk1dCUl0h21/r18MYbUFxc\nc/sUkapp1AhmzqxywqhKk5TuMBqasnciNWV3E1G6tg3XlPp+fqBzTKVaGnSihCE1I1mJSETShh7c\nExGRhChhiIhIQpQwREQkIUoYIiKSECUMERFJiBKGiIgkpF49uGdma4GPq7l5K2BdDYaTSjqX9FNf\nzgN0LumquudygLu3TmTFepUwdoeZFST6tGO607mkn/pyHqBzSVe1cS5qkhIRkYQoYYiISEKUML43\nKdUB1CCdS/qpL+cBOpd0lfRzUR+GiIgkRHcYIiKSECUMERFJSINPGGY22MzeN7PlZjY+1fFUlZmt\nMLOFZjbPzArCshZmNt3MloU/m6c6zvKY2QNmtsbMFsWVlRu7Be4Ir9MCM+uTush3VsG5TDCzVeG1\nmWdmQ+Pqrg7P5X0zG5SaqMtnZh3NbKaZLTaz98zs0rC8zl2bXZxLnbs2ZtbYzN42s/nhufw2LO9s\nZm+FMf/dzHLC8kbh9+VhfafdDiLRd7nWxwXIBD4ADgRygPlA91THVcVzWAG0KlN2MzA+/Dwe+L9U\nx1lB7McAfYBFlcUODAX+AxhwJPBWquNP4FwmAFeUs2738O9aI6Bz+HcwM9XnEBdfO6BP+LkpsDSM\nuc5dm12cS527NuGf717h52zgrfDP+wngjLD8HuDC8PNFwD3h5zOAv+9uDA39DqMfsNzdP3T3rcBU\n4MQUx1QTTgQeDj8/DJyUwlgq5O6vAevLFFcU+4nAIx6YDexjZu1qJ9LKVXAuFTkRmOruW9z9I2A5\nwd/FtODun7v73PDz18ASoD118Nrs4lwqkrbXJvzz3RR+zQ4XB44DngrLy16Xkuv1FHC8mX3/Ivlq\naOgJoz3wadz3lez6L1M6cuAlM3vHzMaGZW3c/fPw82qgTWpCq5aKYq+r12pc2EzzQFzTYJ05l7AZ\nozfB/2br9LUpcy5QB6+NmWWa2TxgDTCd4A7oK3ffHq4SH2/puYT1G4CWu3P8hp4w6oP+7t4HGAJc\nbGbHxFd6cD9aJ8dO1+XYQ3cDBwG5wOfArakNp2rMbC/gaeAyd98YX1fXrk0551Inr427F7l7LtCB\n4M7nB7V5/IaeMFYBHeO+dwjL6gx3XxX+XAP8g+Av0RclTQLhzzWpi7DKKoq9zl0rd/8i/AdeDPyN\n75s20v5czCyb4BfsZHd/Jiyuk9emvHOpy9cGwN2/AmYCEYImwKywKj7e0nMJ65sBhbtz3IaeMOYA\nXcJRBjkEHUPTUhxTwsysiZk1LfkMDAQWEZzDqHC1UcA/UxNhtVQU+zRgZDgi50hgQ1zzSFoq045/\nMsG1geBczghHsXQGugBv13Z8FQnbue8Hlrj7n+Kq6ty1qehc6uK1MbPWZrZP+HkP4ASCPpmZwKnh\namWvS8n1OhV4JbwzrL5U9/yneiEY4bGUoC3wN6mOp4qxH0gwomM+8F5J/ATtlDOAZcDLQItUx1pB\n/FMImgO2EbS9nldR7AQjRO4Kr9NCIC/V8SdwLo+GsS4I//G2i1v/N+G5vA8MSXX8Zc6lP0Fz0wJg\nXrgMrYvXZhfnUueuDdATeDeMeRFwXVh+IEFSWw48CTQKyxuH35eH9QfubgyaGkRERBLS0JukREQk\nQUoYIiKSECUMERFJiBKGiIgkRAlDREQSooQhUg4zmxX+7GRmZ9Xwvn9d3rFE0p2G1YrsgplFCWY1\nHVaFbbL8+7l9yqvf5O571UR8IrVJdxgi5TCzkllBbwKODt+ZcHk4+dstZjYnnLjuf8P1o2b2uplN\nAxaHZc+Gk0K+VzIxpJndBOwR7m9y/LHCJ6VvMbNFFrzj5PS4fcfM7Ckz+6+ZTd7dWUdFqiOr8lVE\nGrTxxN1hhL/4N7h7XzNrBLxpZi+F6/YBDvNgWmyAMe6+PpzGYY6ZPe3u481snAcTyJX1PwST4fUC\nWoXbvBbW9QYOBT4D3gSOAt6o+dMVqZjuMESqZiDBvEnzCKbJbkkw3xDA23HJAuDnZjYfmE0wCVwX\ndq0/MMWDSfG+AF4F+sbte6UHk+XNAzrVyNmIVIHuMESqxoBL3P3FHQqDvo5vynwfAETc/VszixHM\n7VNdW+I+F6F/u5ICusMQ2bWvCV7tWeJF4MJwymzM7JBwpuCymgFfhsniBwSv0iyxrWT7Ml4HTg/7\nSVoTvPY1LWZKFQH9L0WkMguAorBp6SHgdoLmoLlhx/Nayn8F7gvABWa2hGDW09lxdZOABWY2193P\njiv/B8H7DeYTzLB6lbuvDhOOSMppWK2IiCRETVIiIpIQJQwREUmIEoaIiCRECUNERBKihCEiIglR\nwhARkYQoYYiISEL+HyV3nbNSd7QBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f34ea964a10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.plot(plot_loss,'r.')\n",
    "plt.plot(plot_loss,'b--')\n",
    "plt.legend([\"1-Layer LSTM\",\"Multi-Layer LSTM\"])\n",
    "plt.ylabel(\"loss\")\n",
    "plt.xlabel(\"iteration\")\n",
    "plt.title(\"Loss During Training\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train an RNN with MSCOCO Captions \n",
    "Now we are going to try a slightly more complicated example. Let's use the [Microsoft Common Objects in Context](http://mscoco.org/) (MSCOCO) image captions to train an RNN. The cell below shows one way to read, format, and feed the data into TensorFlow. First, we will read the caption file, then we will remove the punctuation, and then train. Due to time constraints, we won't use the full dataset for this training. However, it would be easy to change this and train with more or the entire dataset. Can you see an easy way to do this?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_steps=20\n",
    "## Read Training files\n",
    "with open(\"/data/mscoco/captions_train2014.json\") as data_file:\n",
    "         data=json.load(data_file)\n",
    "\n",
    "TotalNumberofCaptions=len(data['annotations'])\n",
    "\n",
    "sentences=[]\n",
    "\n",
    "##Create a list of all of the sentences.\n",
    "for i in range(TotalNumberofCaptions):\n",
    "        sentences+=[re.sub('[^A-Za-z0-9]+',' ',data['annotations'][i]['caption']).lower()]\n",
    "\n",
    "TotalWordList=[]\n",
    "for i in range(TotalNumberofCaptions):\n",
    "        TotalWordList+=re.sub('[^A-Za-z0-9]+',' ',data['annotations'][i]['caption']).lower().split()\n",
    "\n",
    "#Determine number of distint words \n",
    "distintwords=collections.Counter(TotalWordList)\n",
    "#Order words \n",
    "count_pairs = sorted(distintwords.items(), key=lambda x: (-x[1], x[0])) #ascending order\n",
    "words, occurence = list(zip(*count_pairs))\n",
    "DictionaryLength=occurence.index(4) #index for words that occur 4 times or less\n",
    "words=['PAD','UNK','EOS']+list(words[:DictionaryLength])\n",
    "word_to_id=dict(zip(words, range(len(words))))\n",
    "#Tokenize Sentence\n",
    "Tokenized=[]\n",
    "for full_words in sentences:\n",
    "        EmbeddedSentence=[word_to_id[word] for word in full_words.split() if word in word_to_id]+[word_to_id['EOS']]\n",
    "        #Pad sentences that are shorter than the number of steps \n",
    "        if len(EmbeddedSentence)<num_steps:\n",
    "            b=[word_to_id['PAD']]*num_steps\n",
    "            b[:len(EmbeddedSentence)]=EmbeddedSentence\n",
    "        if len(EmbeddedSentence)>num_steps:\n",
    "            b=EmbeddedSentence[:num_steps]\n",
    "        if len(b)==EmbeddedSentence:\n",
    "            b=EmeddedSentence\n",
    "        b=[word_to_id['UNK'] if x>=DictionaryLength else x for x in b] #turn all words used 4 times or less to 'UNK'\n",
    "        #print(b)\n",
    "        Tokenized+=[b]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[u'a very clean and well decorated empty bathroom', u'a panoramic view of a kitchen and all of its appliances ', u'a blue and white bathroom with butterfly themed wall tiles ', u'a panoramic photo of a kitchen and dining room', u'a graffiti ed stop sign across the street from a red car ', u'a vandalized stop sign and a red beetle on the road', u'a bathroom with a border of butterflies and blue paint on the walls above it ', u'an angled view of a beautifully decorated bathroom ', u'the two people are walking down the beach ', u'a sink and a toilet inside a small bathroom ']\n",
      "[[3, 142, 508, 9, 619, 415, 276, 57, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [3, 4027, 171, 5, 3, 61, 9, 317, 5, 155, 612, 2, 0, 0, 0, 0, 0, 0, 0, 0], [3, 59, 9, 21, 57, 8, 2922, 1963, 136, 1225, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0], [3, 4027, 162, 5, 3, 61, 9, 461, 43, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [3, 638, 6775, 148, 58, 241, 6, 24, 96, 3, 47, 128, 2, 0, 0, 0, 0, 0, 0, 0], [3, 2539, 148, 58, 9, 3, 47, 7524, 4, 6, 84, 2, 0, 0, 0, 0, 0, 0, 0, 0], [3, 57, 8, 3, 2984, 5, 5243, 9, 59, 1228, 4, 6, 486, 240, 27, 2, 0, 0, 0, 0], [14, 4659, 171, 5, 3, 3647, 415, 57, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [6, 15, 18, 19, 55, 31, 6, 75, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [3, 115, 9, 3, 82, 160, 3, 36, 57, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]\n"
     ]
    }
   ],
   "source": [
    "print(sentences[:10])\n",
    "print(Tokenized[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3, 142, 508, 9, 619, 415, 276, 57, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      "[142, 508, 9, 619, 415, 276, 57, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      "Number of words in this dictionary  8768\n"
     ]
    }
   ],
   "source": [
    "############################################# Parameters #####################################################\n",
    "\n",
    "num_hidden=256\n",
    "num_steps=20\n",
    "dict_length=len(words)\n",
    "batch_size=4\n",
    "num_layers=1\n",
    "\n",
    "## Create labels\n",
    "Label=[]\n",
    "for caption in Tokenized:\n",
    "    Label+=[caption[1:]+[word_to_id['PAD']]]\n",
    "\n",
    "NumberofCasestoEvaluate=20\n",
    "TrainingInputs=Tokenized[:NumberofCasestoEvaluate]\n",
    "LabelInputs=Label[:NumberofCasestoEvaluate]\n",
    "\n",
    "#Print out some variables \n",
    "print(TrainingInputs[0])\n",
    "print(LabelInputs[0])\n",
    "print(\"Number of words in this dictionary \", len(words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create our input queue\n",
    "def data_input_queue(TrainingInputs, LabelInputs, num_steps):\n",
    "    train_input_queue = tf.train.slice_input_producer(\n",
    "                                    [TrainingInputs, LabelInputs],\n",
    "                                    shuffle=True)\n",
    "\n",
    "    ##Set our train data and label input shape for the queue\n",
    "    TrainingInput=train_input_queue[0]\n",
    "    LabelInput=train_input_queue[1]\n",
    "    TrainingInput.set_shape([num_steps])\n",
    "    LabelInput.set_shape([num_steps])\n",
    "    min_after_dequeue=100000\n",
    "    capacity = min_after_dequeue + 3 * batch_size \n",
    "    #input_x, target_y\n",
    "    train_x, train_y = tf.train.batch([TrainingInput, LabelInput],\n",
    "                                                 batch_size=batch_size ,\n",
    "                                                 capacity=capacity,\n",
    "                                                 num_threads=4)\n",
    "    return train_x, train_y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you are ready to train an RNN with the MSCOCO captions. Before you start training define the dropout values you want to use for training. If you need a hint check [here](#answer3 \"Any value greater than 0 and less than or equal to 1 will work. For exampe input_keep_prob=1.0 and output_keep_prob=1.0 can be used\")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iteration:  0 loss:  9.76574\n",
      "iteration:  100 loss:  0.629486\n",
      "iteration:  200 loss:  0.459927\n",
      "iteration:  300 loss:  0.179445\n",
      "iteration:  400 loss:  0.238892\n",
      "iteration:  500 loss:  0.175308\n",
      "iteration:  600 loss:  0.185177\n",
      "iteration:  700 loss:  0.183901\n",
      "iteration:  800 loss:  0.17512\n",
      "iteration:  900 loss:  0.187115\n",
      "iteration:  1000 loss:  0.0794678\n",
      "iteration:  1100 loss:  0.213151\n",
      "iteration:  1200 loss:  0.178427\n",
      "iteration:  1300 loss:  0.150617\n",
      "iteration:  1400 loss:  0.18845\n",
      "Done Training\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "loss_mscoco=[]\n",
    "#######################################################################################################\n",
    "NumberofCasestoEvaluate=100\n",
    "TrainingInputs=Tokenized[:NumberofCasestoEvaluate]\n",
    "LabelInputs=Label[:NumberofCasestoEvaluate]\n",
    "\n",
    "\n",
    "## Make Variables\n",
    "# tf Graph input\n",
    "x = tf.placeholder(dtype=tf.int32, shape=(batch_size , num_steps))\n",
    "y = tf.placeholder(dtype=tf.int32, shape=(batch_size,  num_steps))\n",
    "#tf.get_variable_scope().reuse_variables()\n",
    "\n",
    "variables_dict = {\n",
    "    \"weights_mscoco\":tf.Variable(tf.truncated_normal([num_hidden,dict_length],\n",
    "                                                     stddev=1.0,dtype=tf.float32),name=\"weights_mscoco\"),\n",
    "    \"biases_mscoco\": tf.Variable(tf.truncated_normal([dict_length],\n",
    "                                                     stddev=1.0,dtype=tf.float32), name=\"biases_mscoco\")}\n",
    "\n",
    "\n",
    "# Create input data\n",
    "train_x, train_y =data_input_queue(TrainingInputs, LabelInputs, num_steps)\n",
    "mscoco_dict=words\n",
    "X_one_hot=tf.nn.embedding_lookup(np.identity(dict_length), x) #[batch,num_steps,dictionary_length][2,6,7]\n",
    "y_one_hot=tf.unstack(tf.nn.embedding_lookup(np.identity(dict_length), y),num_steps,1) #[batch,num_steps,dictionary_length][2,6,7]\n",
    "y_target_reshape=tf.reshape(y_one_hot,[batch_size*num_steps,dict_length])\n",
    "\n",
    "input_keep_prob=1.0\n",
    "output_keep_prob=1.0\n",
    "\n",
    "#Create a multilayer RNN\n",
    "\n",
    "layer_cell=[]\n",
    "for _ in range(num_layers):\n",
    "    lstm_cell = tf.contrib.rnn.LSTMCell(num_units=num_hidden, state_is_tuple=True)\n",
    "    ############# add dropout #########################\n",
    "    lstm_cell = tf.contrib.rnn.DropoutWrapper(lstm_cell,\n",
    "                                          input_keep_prob=input_keep_prob,\n",
    "                                          output_keep_prob=output_keep_prob)\n",
    "    layer_cell.append(lstm_cell)\n",
    "\n",
    "cell = tf.contrib.rnn.MultiRNNCell(layer_cell, state_is_tuple=True)\n",
    "outputs, last_states = tf.contrib.rnn.static_rnn(\n",
    "    cell=lstm_cell,\n",
    "    dtype=tf.float32,\n",
    "    inputs=tf.unstack(tf.to_float(X_one_hot),num_steps,1))\n",
    "\n",
    "output_reshape=tf.reshape(outputs, [batch_size*num_steps,num_hidden]) #[12==batch_size*num_steps,num_hidden==12]\n",
    "pred=tf.matmul(output_reshape, variables_dict[\"weights_mscoco\"]) +variables_dict[\"biases_mscoco\"]\n",
    "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y_target_reshape))\n",
    "optimizer = tf.train.AdamOptimizer(0.01).minimize(cost,aggregation_method = tf.AggregationMethod.EXPERIMENTAL_TREE)\n",
    "\n",
    "\n",
    "init_op = tf.group(tf.global_variables_initializer(),\n",
    "                   tf.local_variables_initializer())    \n",
    "\n",
    "\n",
    "with tf.Session() as sess:\n",
    "        \n",
    "        sess.run(init_op)\n",
    "        coord = tf.train.Coordinator()\n",
    "        threads = tf.train.start_queue_runners(coord=coord)\n",
    "\n",
    "        for i in range(1500):\n",
    "            x_input,y_input=sess.run([train_x, train_y])\n",
    "            loss,_,y_target,y_pred=sess.run([cost,optimizer,y_target_reshape,pred],feed_dict={x:x_input,y:y_input})\n",
    "            loss_mscoco.append([loss])\n",
    "            if i% 100==0:\n",
    "                print(\"iteration: \",i, \"loss: \",loss)  \n",
    "        print(\"Done Training\")\n",
    "        coord.request_stop()\n",
    "        coord.join(threads)\n",
    "        sess.close()    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input Sentence\n",
      "[u'the', u'vanity', u'contains', u'two', u'sinks', u'with', u'a', u'towel', u'for', u'each', 'EOS', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD']\n",
      "Target\n",
      "[u'vanity', u'contains', u'two', u'sinks', u'with', u'a', u'towel', u'for', u'each', 'EOS', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD']\n",
      "Predicted words\n",
      "[u'sky', u'contains', u'two', u'sinks', u'with', u'a', u'towel', u'for', u'each', 'EOS', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD']\n"
     ]
    }
   ],
   "source": [
    "#Lets look at one input data point and its prediction\n",
    "print(\"Input Sentence\")\n",
    "batch_element=2\n",
    "print([words[ind] for ind in x_input[batch_element,:]])\n",
    "print(\"Target\")\n",
    "print([words[ind] for ind in y_input[batch_element,:]])\n",
    "print(\"Predicted words\")\n",
    "print([words[ind] for ind in np.argmax(y_pred[batch_element::batch_size],1)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does the single word prediction cell above represent a deployment scenario? \n",
    "\n",
    "No, the ground truth is being propagated through the network. The  predicted caption above represents a single predicted word based on the previous ground truth.\n",
    "\n",
    "In the classification part we had a function for our network. If you were going to make a function for this network, what would you include?  What would your inputs and outputs be?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Free our GPU memory before proceeding to the next part of the lab\n",
    "import os\n",
    "os._exit(00)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## References \n",
    "[1] Imanol Schlab. TensorFLow Input Pipeline Example. http://ischlag.github.io/2016/06/19/tensorflow-input-pipeline-example/\n",
    "\n",
    "[2] Denny Britz. Practical Examples for RNNs in TensorFlow https://github.com/dennybritz/tf-rnn\n",
    "\n",
    "[3]Lin, Tsung-Yi, et al. \"Microsoft coco: Common objects in context.\" European Conference on Computer Vision. Springer International Publishing, 2014."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}