demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# An Introduction to Recurrent Neural Networks\n",
    "\n",
    "Recurrent neural networks (RNNs) have become increasingly popular in machine learning for handling sequential data. In this tutorial, we will cover [background](#1.-Background) about the architecture, a toy [training example](#2.-Training-a-toy-model), and a demo for [evaluating a larger pre-trained model](#4.-Evaluating-a-pre-trained-model).\n",
    "\n",
    "**Date**: June 26, 2019\n",
    "\n",
    "**Authors**: Jennifer Hu (MIT), Ben Huh (IBM), Peng Qian (MIT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Prerequisites\n",
    "\n",
    "To run this tutorial, you will need to install [Pytorch](https://pytorch.org/). Just follow the instructions in the docs to install the library and its dependencies. To test your installation, go ahead and run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will also need [numpy](https://www.numpy.org/), which comes automatically installed with a scientific Python distribution such as [Anaconda](https://www.anaconda.com/). Of course, you can also install it manually. Test your installation by running the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Background: What is an RNN?\n",
    "\n",
    "Note: The following figures were taken from excellent blog posts by [Christopher Olah](http://colah.github.io/posts/2015-08-Understanding-LSTMs/), [Jianqiang Ma](https://medium.com/@jianqiangma/all-about-recurrent-neural-networks-9e5ae2936f6e), and [Shi Yan](https://medium.com/mlreview/understanding-lstm-and-its-diagrams-37e2f46f1714).\n",
    "\n",
    "In both machine learning and everyday life, many tasks make use of **sequential** data. Take language understanding, for example. As you comprehend each word in this sentence, you draw upon information from the previous words. RNNs have an **inductive bias** for handling this type of data, as parameters are shared across positions in the sequence.\n",
    "\n",
    "How does this architecture compare to \"normal\" neural networks? Let $A$ be a neural network, $x_t$ the input, and $h_t$ the output or hidden state vector. Consider the following diagram (credit to [Christopher Olah](http://colah.github.io/posts/2015-08-Understanding-LSTMs/)):\n",
    "\n",
    "<img src =  \"img/rnn_unrolled.png\"     style=\"text-align:center; width: 500px;\"/>\n",
    "\n",
    "\n",
    "The self-loop at $A$ represents the **recurrence** of the network. We can \"unroll\" it into multiple copies of the same network $A$, sending information down the temporal chain."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Training a toy RNN\n",
    "\n",
    "Let's try training our own RNN model.\n",
    "Here, we implement the model from the following paper (original Tensorflow code [here](https://github.com/gyyang/multitask)):\n",
    "\n",
    "> \"Task representations in neural networks trained to perform many cognitive tasks.\" Guangyu Robert Yang, Madhura R. Joglekar, H. Francis Song, William T. Newsome & Xiao-Jing Wang (2019). [*Nature Neuroscience* Volume 22, pp. 297–306](https://www.nature.com/articles/s41593-018-0310-2).\n",
    "\n",
    "This will train a RNN model for 20 types of cognitive tasks. Here's an example: \n",
    "\n",
    "<img src =  \"img/XJW.png\"     style=\"text-align:center; width: 450px;\"/>\n",
    "\n",
    "\n",
    "We first implement a RNN model. `RNN_multitask/RNN_rate_dynamics.py` is a custom RNN implementation of **continuous-time rate-neuron network dynamics**, which is commonly used in neuroscience models:\n",
    "\n",
    "$$ \\tau \\dot{h} = - h + \\sigma(W_{hh} h + W_{ih} s) $$\n",
    "\n",
    "where $x$ is neural (hidden) state,  $s$ is (sensory) input, and $W_{hh}, W_{ih}$ are synaptic weight parameters (recurrent and input weights).\n",
    "\n",
    "Note that this model uses continuous-time dynamics description, $\\dot{h} = f(h(t))$, \n",
    "whereas deep-learning models use discrete-time dynamics: $h_{t+1} = f(h_{t})$.\n",
    "\n",
    "The above dynamics can be simulated via Euler-like integration: \n",
    "$$\n",
    "h_{t+\\Delta t} = \\gamma h_t + (1 - \\gamma) \\sigma(W_{hh} h_t + W_{ih} s_t)\n",
    "$$\n",
    "where $\\gamma = exp(- \\Delta t/\\tau) \\approx 1 - \\Delta t/\\tau$ is the decay factor, and $\\Delta t$ is the discrete timestep,\n",
    "which is implemented in `RNN_multitask.RNN_rate_dynamics.RNNCell` as\n",
    "\n",
    "```\n",
    "activity = self.nonlinearity(input @ self.weight_ih.t() +  hidden @ self.weight_hh.t() + self.bias)\n",
    "hidden   = self.decay * hidden + (1 - self.decay) * activity\n",
    "```\n",
    "\n",
    "Here's a sample code for running the RNN model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "from RNN_multitask.RNN_rate_dynamics import RNNLayer\n",
    "\n",
    "T, batch = 1000, 100\n",
    "n_input, n_rnn, n_output = 10, 500, 5\n",
    "\n",
    "rnn  = RNNLayer(n_input, n_rnn, nn.ReLU(), 0.9, True)   # input_size, hidden_size, nonlinearity, decay, bias\n",
    "\n",
    "x       = torch.randn(T, batch, n_input) # input \n",
    "hidden0 = torch.zeros(1, batch, n_rnn)   # initial state\n",
    "output, hidden = rnn(x, hidden0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main training code is implemented in the file `RNN_multitask/multitask/train.py`. Below is a sample code for training the model for the 'Mante' task, which trains for mixed motion and color cues:\n",
    "\n",
    "<img src =  \"img/Mante_task.png\"     style=\"text-align:center; width: 350px;\"/>\n",
    "\n",
    "Over time, the cost should go down and performance should go up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trial       0  | Time 0.00 s  | Now training contextdm1 & contextdm2\n",
      "contextdm1     | cost 0.601940| c_reg 0.000000  | perf 0.26\n",
      "contextdm2     | cost 0.712529| c_reg 0.000000  | perf 0.16\n",
      "Trial   32000  | Time 50.96 s  | Now training contextdm1 & contextdm2\n",
      "contextdm1     | cost 0.213876| c_reg 0.000000  | perf 0.53\n",
      "contextdm2     | cost 0.220911| c_reg 0.000000  | perf 0.50\n",
      "Trial   64000  | Time 103.40 s  | Now training contextdm1 & contextdm2\n",
      "contextdm1     | cost 0.156858| c_reg 0.000000  | perf 0.66\n",
      "contextdm2     | cost 0.176406| c_reg 0.000000  | perf 0.59\n",
      "Trial   96000  | Time 156.09 s  | Now training contextdm1 & contextdm2\n",
      "contextdm1     | cost 0.122628| c_reg 0.000000  | perf 0.77\n",
      "contextdm2     | cost 0.161079| c_reg 0.000000  | perf 0.66\n",
      "Trial  128000  | Time 208.24 s  | Now training contextdm1 & contextdm2\n",
      "contextdm1     | cost 0.110657| c_reg 0.000000  | perf 0.81\n",
      "contextdm2     | cost 0.118822| c_reg 0.000000  | perf 0.79\n",
      "Trial  160000  | Time 260.23 s  | Now training contextdm1 & contextdm2\n",
      "contextdm1     | cost 0.128146| c_reg 0.000000  | perf 0.76\n",
      "contextdm2     | cost 0.071062| c_reg 0.000000  | perf 0.88\n",
      "Trial  192000  | Time 311.55 s  | Now training contextdm1 & contextdm2\n",
      "contextdm1     | cost 0.155235| c_reg 0.000000  | perf 0.78\n",
      "contextdm2     | cost 0.102100| c_reg 0.000000  | perf 0.83\n",
      "Optimization interrupted by user\n",
      "Optimization finished!\n"
     ]
    }
   ],
   "source": [
    "from RNN_multitask import multitask\n",
    "\n",
    "hp, log, optimizer = multitask.set_hyperparameters(model_dir='debug', hp={'learning_rate': 0.001}, ruleset='mante') #, rich_output=True)\n",
    "run_model          = multitask.Run_Model(hp, RNNLayer)\n",
    "multitask.train(run_model, optimizer, hp, log)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's visualize the results. The analysis functions are in `multitask/standard_analysis.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAEICAYAAABhxi57AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztvXmUZVd15vltpZQIpeYBEEpQMAhoGWwBCRaTwYCNwBgom8YM1S3TsvVHg40Lu4woqiiKwqtxdbcxtUzhUjEJNwZkDIVMy8YgxAxqpRAySEIoJQcoheYJDSVSqTz9x7tf3t97uW9GZMTLeBmh/a0V6544994z3/vu95199onWmgqFQqGw8thv1gUoFAqFByrqBVwoFAozQr2AC4VCYUaoF3ChUCjMCPUCLhQKhRmhXsCFQqEwI9QLuDAVRMSlEfG8WZejUFhNiLIDLqwWRMRHJG1trf3bWZelUJgG6gu4UCgUZoR6ARemgoiYj4gXRsQ7IuLsiPhoRNzZSRObJq57a0RcFhG3RcSHI+LA7txvR8TXJ9JtEfHYiDhd0usk/XFE3BURf7eyNSwUpo96ARf2Bl4m6ROSDpd0jqS/mDj/OkkvkvQYSY+TtKCk0Fo7U9LHJP2n1trBrbVfn2qJC4UZoF7Ahb2Br7fWzm2t3S/pryT9wsT5v2itXdNau1XSn0h6zYqXsFDYB1Av4MLewPUI3yPpwIjYH3HXIPwjSQ9fkVIVCvsY6gVcmAUegfAjJf2kC98t6SCfiIiHTdxXJjuFNYV6ARdmgTdExMaIOFLS2yR9sou/RNLPRcRJ3cTcOybuu0HSo1eumIXC3kW9gAuzwF9L+kdJV0u6StK7JKm19kNJ75T0RUlXSvr6xH0flHRiRNweEf995YpbKOwd1EKMwooiIuYl/U5r7YuzLkuhMGvUF3ChUCjMCPUCLhQKhRmhJIhCoVCYEeoLuFAoFGaE/Re+ZHo4MKId3IW3dcf7cP4QhH3+fsStQ/je7rgBcXchvL478hdmG8IHJPGMC4T3S86z3OuSOObbFjj/oO64feC88TOEWe8dyXmWP4t7EML/I0mXbUWO5AHDfmG9HpykuR5hl5H3cxC6XOzLByPsPuD9DDt95pmNgR2IY/pMK+OGvM9lYb+xX37aHQ8cKItx30DYzwr7dXtynmW6G2GPS7Zv9gzwfoLP2/YFzjtd1pXpuj+G+tVtzfa/U7q5tXbMQPEWxCmnnNJuvvnmRV170UUXfb61dspS81oqVvQFfLBGTgKkfikUl0w9B+Fru+MdiOML+sru+BTEXYCwl1YdhLitCD8UYa8CoNU/HxQ/VFyudX1y/ibEMd97k/N8Qdiw9XbEcSAb8wiz3k5/y8D99ydxNKb9HsK/2B1/lNwvjZw7SNKdiPsJwk/sjpch7jiE55P7j0R4rjuyL09E2P3G+xm+ujtypQf7wvW6F3FMn2nxZWjwBefxcgvinoawzTwej7jsZcz2uwHhZ3fHKxF3K8LP7I6sy4UIe1xyrHMJost/D+LY1+wXj12+VA9D2HU4AXEsl/uDdoVsd+fLZ+D88WG4x7j55pu1efPmRV0bEUcvJ6+lYkVfwIVCobByaMq/3fcdrOgLeO6pc/rQ5nd0//kb8gm4gr+pZ3fH30QcSc9V3ZHftYS/VV6EOF7L31o3A+nK6xD+Snfk9wvJlO97FuLOQfgxSfnmEP58UqY3I3xFd+SP9A8Q/kaSP9N6QXc8GHFXIcy6OI/5gbI63ycijtdmILfwd9mliOMYcPrXIu4ohF3Hs5M4qf/euwJxrOtzuyPbh9/oH0XYfIDtxrDH08OSOEl6qnYFx4VfDsyf6R+enL8IYY/bxyKOY+287shx8xWEH5+c5zPAeLcrOSlfH06XIgzHiL+HOdZuT67tx3XEH2h5aBr/Dt/3UF/AhUJhjaK+gAuFQmFGqBdwoVAozAj7/gt4RRdiHBbRrNZ55vVBA9d6NpWzrrQ8sBrLWd1HJ9fyfs6Gc+bZ1g20suBss8vK2eSbkmtpmUAF06oZZ5gvfDf+8QnKVSyAK0s7uOMRviE5z+l8JecPR5jT/Zd0Rxr/sGFcVjbgsxH+ZnekFMi07kvi5pMyUkrldHlm70T4ftafg8Q2XWw/mrdkpio/VQ4PLg4smrd4EFDC5iBwGVCXF71+1+QJNpuHC5XUJyHsZ4BWFLRSuCE5T8/5bGJXkVYaHE4uA4cKzef8bF6fxLEMrPMXpYtaa5u0RGza9PNt8+ZzF3VtxCOWlddSUV/AhUJhDWPf/gKuF3ChUFij2KHx7/B9DzN7AZu+kPGdhLCZMJkyKY+pDA3jydptuEMa9V2EaThkVvx9xNEI3fRrSCGwMc83EfdMhM2GyVTHtql0wuRvmXUdqf7nEHYFsmVQPD8kURyWxA9RfPNGShC0/r964jppnJZ71QI7nlZo5qKUCKjnOH2Wn51puSHTo6R+QH0JcWwr8nm3F1d10LrN/XVQEif1deRgYVmtv2G5AdUg03F264FJmGpP5uOTC0FYfMsRrDKblePd3f3ogWvdxSx/li6HIJ83NxuHxfKx72vA9QVcKBTWMOoFXCgUCjNAfQGP4QD1s7hmZWR/VyNsKsNJY1IiUxquEeKa/AyUMzgxboZJqsZymcFm/hWkfmaYTJW+DAxSti9BYtjptAWLkLjeyD4DDj+/jyOVc7twTT/Pm97RmGCM6oEi24cDZ9u5IP/4rtycDad1iO+76fI+jm15SxJ35CV92HT6Etz/RIQzXw7Ho7FMhb8Dfsvy3dO1MduCtJnOatyHJyJ9tss9XbvdjfbjLL7zpV+MJ0C6obJikOJ/K4njuHNZOJZZr/nuyHFHtcTjaqx9EM6MQ6jmEFaxmBfL4nzR1WNWQ06fssXyUS/gQqFQmBF2qJYiFwqFwsxQX8A7sV092/XMLmdw703CQ+dNlThDTAnAtIyUkRIF403XSb9ItUzhSZ9osO46UaKgG5JsYv5rCJu2kl4Svp9UmXU1lRzyges2pCvAIbnGbJt12YiwaTPlnG8jbGmB9JXl+k6SJ63fXdd7kjipp6iUMNgvmbtK5ul2o0uaX0SYcoFBIwbm6zbmuGS7eAkA2+JbCHsMZu4ypX7cMX3KBZmEQGnJljw0vKAVhOvFsUDZgIY4zotp8T6vNaF0RkMZj322ZWYlweeKZV0aSoIoFAqFGWHffwEvakuiiDg8Ij4VET+IiMsj4hkRcWREfCEiruyOR+ztwhYKhcLi4RfwYv52j4j4UETcGBHfHzgfEfGfI2JLRPxTRDwlu24Si90T7r2S/qG19gSNWMLlks6QdF5r7QSNHI+esci0CoVCYQUwvRewpI9I2t2WRS/WaG3LCZJOl/T+xSS6oAQREYdJ+iVJvy1JrbVtkrZFxMslPa+77CxJX5b0lt2ltUO9Zpvto0YzMOt2dNZDfc4uqKlV0jTK5mnUcrP9vKR8pdqG5Fouzhozg+qO1CppYmStj5oaNU63xZAW+MLuSN1yHmGvIJzYT2sn/Ct7KOJobsV2c3tljlKkXrcb2mbH7fKTBc6zrDQ/tE6dLc5j+Yb85zj9IQc0dvdNLXLo2mz1VrZojm2V7aFD7Z3avccIn4GDkjDPZ9sXDa2U8xgeWgDp8vNTjenTB5FXdvJ+5uWxz7LMIextr1h/trs1ZI7F5WN6Dtlba1+NiLndXPJySR9tI+9m3+5Ug2Nba9ftLt3FfAE/SqMx/uGIuDgiPhARGyQ9FIlfr/G5gJ2IiNMjYnNEbN6WXVAoFAp7BXv0BXy031Pd3+l7mNlxGv8G26rxZQopFjMJt79GP5K/11q7ICLeqwm5obXWIiL1a9laO1PSmZJ06MA1hUKhMH00DdsW7YKb91V3lFslbW2teWHQpzR6Ad/gT+yIOFbSjQsl9CD1dM1OQ+i/lE1lSjK2Cgthywn0fcKfn8zMirR2Q3Ke+ZPiOi1KGKSFpmWsC6lcZlI3j7CpHM21SEVtzkSfNWO+hZP0ac7jdhna8XdsVVpyniZ7LhfLemsSztpP6uvwecSxD02jSG8pMbgObF9SbPvYoTkY6505e2H+9NFjaYflZ7ksI9HMLRtXJMHcgdptwXFLOJ4SAU23XO45xFGm8zNEuYvSkccQ249jnG6MLQ2w/XifpQVKfpQP3S4cS6TMtyZxy8eKWkFcq3GFZaPyxY5jWFCCaK1dL+maiHDfvUCjfj5H0qld3KmSPrsnpS0UCoW9j6lNwi2EcyT9r501xMmS7lhI/5UWbwf8e5I+FhHrNZozeb1GL++zI+I0jT5UX7W0chcKhcLewPSWIkfExzUyOjg6IrZK+vfqSEBr7S81IqovkbRFI9Lx+jylcSzqBdxa+67GFywZL0jiBrFNPa0yXZ/H+TmEfR2/6Ul1Hc8Z9HuS86RsTJ/wbOzQiibTJtLujBbTLS4pvPMlPSQV88xxtvMP8x1yquL7mCdpsdvyaYjjiiVuak5qb7CuzoNtwZVodllMKkqX2PbJTKqZyR1cBcV6u18om5DCuy1IjzlGLLFkjp2k8ZVgptDsK/ahpYELkjjmwfKx3E6Lk9OUANxfLCslEJeL1i2Zw6mhbYDcVnxGhqQjj4Ehh07m2r8wcN79wbI8PDk/fSuI6UgQrbXXLHC+SXrDnqZbK+EKhcIaxb6/Eq5ewIVCYQ2jXsA7sV49HTcVeQ7OZw5wSM9o5G3aRHpI2m0qRPpJKswth+xghPTtKoS3dEfSP1pUZJYDlBP2m7hOyiUIpp/NRtNwnRTb5Wf+pPimerQoeRHCXOBh2jnkdMXSDtNnW1g6IFVluV1H0n5KS243Sk/zCFsWWMi4iDP4mUOmOwfOs40s03AMnoywxwjHDSUc9/vQjkceD6wr73cZOVOebenD8tNxksc75Spe674akh1oxJpZjzDd+e7ItuAYtgUTn0eOd9eF5V8+6gu4UCgUZoR6ARcKhcKMUA7ZC4VCYYaoL+Cd+Jl6PTDbhprhnfugIY7hbFt6mgBZP6OzbaZPMyfrxVxJwvusVfEe6prWt05CHDVkapxG5liIcUzrB0kcTd6oNxuZgx1qbnSizm+ETA+ndp2ZCVFXtPZK0yvqlpnzemqclyXnqcG6fOx3apVOi1omncpYj2X61F2pLbvc1Eh/oF1BDTfb04zty7Q8RodWe9qskm1J3dX5UoOnKabz5bjnWPS4Z1sQ30vi2Fccd17p9pOB89kGCVw1mDnSXz5KgigUCoUZoV7AhUKhMCPUC3gMTb1piqnKHM7TUYhpGekvzaxMFWkudX1yLd3XcxXQkxE2beY+bszLtItUk2ZWzpf0lPTKtIq0mavPTBUpMdAcx3ID60eJwLSa9SNtdRuRXmdtzbISpKiZXMLzmYRAacimgqSiC7U1x0BWPt7vtqQZGtvNfcDVY6T1RLaqLis3ZQdeazlgyNTSZc0cO0l9G1LCYLvclJxnXTxGOBYogWT767F82WrJoVV5WfqUOywxUMajNOL+GtqrcOlYtDe0maC+gAuFwhpFWUEUCoXCjFASxBjWq6copuukIfMImz5xVpdUzzO4lBiyGVpStiH65NleUjHKBZYTSI+4osdbzFNCyFZaMc2MVpKKkr69vDsOUV23Fe+h72CXe8jHLmmjJSL6ciXseIcSDOviLX2YPq0QstVf7DdT6Exi0UAcw5aRaK3BVXtuN/bP0LhwumzrzGf10E4vRyVxlFMsGXEszSPscce2ZF1t/UDHSFsQdr1ZjkzGY1tRLmJZXIahtFwvpsXnJbNqYr/4vrKCKBQKhTWBegEXCoXCDFEv4J24W/0suA3OSUky/6GkObz26iQuM5LPHL1I47O9pvakctlsMWkzqZIXbbCsDM93R9LbVyLMfA06yDFV5Qw3Z/FdlyF/wo5nnYbg9swWxUj9jD63FKLc4d1zv4C4bNddyilsq5drV7AuttgYcjBjSYp+aUnbswVAlCBIi03njxm4diErg+w85QynxXvYFvadzK2uaKnjurItaT1zVHKez8t8d+TiD8ohlKHcbpl1jdQ/Z+wXylSWU4Z2487SXD5qEq5QKBRmhJIgCoVCYUaoF3ChUCjMEPUC3gmaodncaciJuc/T0UimGVHfo1Zm0zLqj5sRpu5mbZcmb5lzeOq+1PKsb2VOd5g+Hej8LsTGW7qKUZ97LsJzzKzDD9Fwvu8hEGuvgjds578eot7d8E6/Aenf0qVL5YwaqLVTOis67mD8091/GNKndu6ysi2pUWZ7/VGXtfbN8rF57GiefUEN2zr8hiRuMl3XleXfAJH1tq2jIzX8DWjj+7s2oBleNk/A8v9Bcp7mXF9H2Hor25IaqrXzTPdlPB3osK4ZuKkB73NZ+IzSZM17xnHugmaZfsZoprd81BdwoVAozAj1Ai4UCoUZoawgBuFmyfbrknpaRUqSOfrInJdI0m90x68hjiY6pEqmgHMDZTUV5B5XmQnOkJmb6RklEmGT66OcGBpgjoVxYsj0cZl9HXj7Y1hZX4vG3pA5RJZ0lK8dGLfHuTKZ7Ze0k/c+ldoQ7Zk6TegocOmjaFvV5f8E2jtR7+jqSAmBy78e8qvd8RKcZ1kzh8DQuTZgQB3lgcEljt/sg0dYo2D60LHWdf12BL0kZfaJ4P23v60P75g4TiS/s9vZfJRuMrBb3dVDpl+UhtzeNO+bR9hVvGPgvOWOaxHHPsz8SE8H5YynUCgUZoA1IkFExLxGH5v3S9reWtsUEUdK+qRGH47zkl7VWrtt7xSzUCgU9hRr5AXc4Zdbazfj/zMknddae3dEnNH9/5bdJRDadRaalIj0JZsNJ9NzPP36Uo74QhLHWVluy27aQ9mAs8Fevfc0xFEuMRPlrC5pncswtl3Ne8hLf647kqCR1891R3YXw3d1Rw42zn271VmCuxBm/Jx2Bfm6+T5asP1NHw7Xi+U7GuGujK9Gb9z193344McleXLY7bwJScLOYH+vxaP9DDwet85jcfw8zs8jnBFyxJ08n5znyKH9ykXdkXoQ3eVY2+jXcD7ibb22ZLWEJJq96nj6c6YMZrmCLUkFxPFMn2OYeXkLLipLJyB8QXd8KeI4mp0vy0eLCKs9meOlpWPffwHvt/Alg3i5pLO68FmSXrH84hQKhcI0sX2RfwsjIk6JiCsiYkv30Tl5/pERcX5EXBwR/xQRL1kozcW+gJukf4yIiyLi9C7uoa2167rw9Ro312ShTo+IzRGxechtX6FQKEwftoJYzN/uERHrJL1P0os1cnPxmog4ceKyfyvp7NbakyW9WtJ/WSjdxUoQz26tXRsRD5H0hYgY83vTWmsR0bIbW2tnSjpTkg6NaJnxucFfA5NKUhY6+rADEcoWlBsscZBQkophMntnWTjDTCOBzIkN6Zm7jz8wmd9YllX635PcnoW4n0N4biInafz3znScNWTXep6cS1FINplXtq9vFgfTgODcuFuGLU/XQonccjApuu+nhEB3Pya7yHP/C3DeJhOULeA6KH4tSZ9fQOwl53Ee4thHbmPez3Q3ThwlaSvCXsHyGztjrtQf7Qx77HNcZYsfhvwZ2+qGO3jz2oX8DRNOax5xmaHK2Yjjc+MyUHhjXn4GOSqXj6lKEE+XtKW1drUkRcQnNFIBuDtXU6+sHKbx7kqxqBdwa+3a7nhjRHymK8wNEXFsa+26iDhW0o2LrUmhUCisCNqizdCOjgh+oZzZfTwax2n8N2erxg0kJekdGikFv6fR79sLF8p0QQkiIjZExCEOS/pVjWaIzpF0anfZqZI+u1BahUKhsKLYscg/6ebW2ib8nZknuFu8RtJHWmsbJb1E0l9FxG7fsYv5An6opM9EhK//69baP0TEhZLOjojTNJqIfdUSClwoFAp7B03TXIdxrcaNozZq3NBDkk6TdIoktda+FREHamQCNKgOLPgC7jSPX0jib5H0ggWLDWzXrg5OuMqHuuum7kitmOqcVclM0yLY/hSuM+fr1MpoYmMhZ2jbeWt12Solqe+l8S23/zPC1gj/FnHUPTsNchskpfW/hPPz3ZHmXjQzcyttTOKkcZO3Djf/XR8+mj/izoNqY6aLfhJx1IM/1h3nEJfs9rftM33U+sfhvPN/30D6zpftx/TdG2wf3s9edntRT6bJmfNgWzKvrqw3/xBRnLex3t2TR5pK7pfEZdvaZ6ZpUj8qqJDzQfa1HOucR6E27PE8hzjycT+7nKehyZu5Ost3Q3Ity/ItLRNtohDLw4WSToiIR2k0iF4t6bUT1/xYo3fiRyLif9LoIblJu0GthCsUCmsTU/wCbq1tj4g3ajSju07Sh1prl0bEOyVtbq2dI+kPJf23iPhXXe6/3VpLjROMegEXCoW1ix0LX7JYtNbOlXTuRNzbEb5M43RwQazoC/jB6veJMi2iv146q/F6IRI2roqzOQvNVrIVP/z+58ob7oNlWje0P91CEoPXLt0+cN73zSFO7/oxLu7C1CjmYChnO6Qxx61f7cM7M0aavNaLw67AeaoRO77Th+e7IzWWAzCKL+nkLHqA2Yp03SG0NzoA591YP4E3Ic4lX9n18thSQlD4h3fhsU3dMDI8CO6F7MaOtXcndtYTcD/tEw9PPDyzXY/sLJBob8VBcsutu5b1RFgteZBjMNJ3sLNiU1CSc1Y0SCTj9spRPkP8IPT9Qw6tGJ7vjjQdY+u4C7+LuOMQdhd8D3G83/Va0G5rTzBdDXivoL6AC4XC2kW9gAuFQmEGaJqqBLE3sKIv4PvUszGzNq7yoZzg2dqhlWz3ThylcRuRbKt2zrryPq9dIuumfYnTzVbvST3Vo61AtmUOFYZb/l0fNm2k72MyZDtboVxDiw23C5kwZ6gf/efj+UyW71D4672l23OGtJTprndjwcjhRuxT4zz44fFITI1ff8+u5w/8qHZBtpJQ6scP22o9G96dgQrcRh+9znPXLEflQ9hjj+nfiAfaaVBhGCtXd6RdBW0zEjfPY++L53RHCERj49nPDmUBlsX9xjrRoZSfN4419jX7wP1KNYfShpub9bsmCdOhFuvtYZVtO7ZkNI2/YPZB1BdwoVBYu6gv4EKhUJgBahJuHKGepnvil65JSJVskD3EIEybhrYB8qINyhIM04NGRiUfk+RFCp9ZX3ACnDAtJX37NMJm8CwTrTRMMbm9UibN/D9JnlIvN5CqMnw7JAT3Dy1GiCO7hqVFCtuFuw/tzAsN4zZmW9FgwjP6nA3PFtjQ4P9efOXs6DgsVQnKKX4eh3bnZV1crpuQfkb3OW5okWC6TenqcwgfnpxnW3g80L0PFyO5DkPb+HhXpmcjjr6D3RasU7ZVl9SPsaEdlOe7I/uK6bpetHyg9OGyTNcZj+oLuFAoFGaC+gIuFAqFGaFewIVCoTAjTNcXxF7Bir6Ad6jXOzO99GSEratR66P+5B82asDUrKxF0ZyMWhpX6ViLYpmoPVtPpX5Gcx1rrGOmVQhbg+RYOAdhamkGdm3fmS71wZ8kYabP8rn81AqpW1JPdhrziKPpkheKZSsBpV4DZhzTd1mod1O7t27I8lNvdrtS46V5nldY0qlLZk5Fx68cN6yXzQbpJIp95XHD+2ky9pSJ66TxceEy0EEO2835c5e5SxB2W3O+gPV6bHcccrjuvuC45zzFtxF+SnKe48nPAOvHZ9ftyjFKs0yPbbbVsp3xSPUFXCgUCjNBLcQoFAqFGaK+gMfh9jA9IeW6CmE75iH9zJyS0GyFpleWC+j3lxuCk847DdJemtNM+jCWxleSmVbRmRDplU2HSBUziYH0l/VyvkMmPJY4uMooo6qkh6SlmVMUtjXlApeR92erx1g+0l6XgeZqlFMsEzH9IWnH2JzEsUykyq5rJldJ421o6YFtkckJXJXIMeQxwPblGHW9voS4w5Mwxwppvc3QmP5jEXZdKKHwfvcL6zSPMMezZQpee2QSppzBfvcYogSRyTGP1xRRX8CFQqEwI9RS5EKhUJgh6gt4PDPTomw2m3KBaSlpeUZFsYgrpehDM8zzCGcz72PuZrsjqSYlDJfrmiRO6mn1+YgjVXT6pGekyG4zUm1SSbfFEBW9ZuI6pimNz9zPTZRJGv+IcL2YFp3lGBz3lANcL95zTxKmoxfKKZZjSIXp7Mb30WKF5fd4WGj1mNRLD/OIY7sZrGvmR5pyFPv11onrpFxiYV/Q2Y0tfHgP0/pKcp7SlqUxjhs+LxyPlrko11AucXtTlmC+bkuWj33odmO/LRtlB1woFAozRL2AC4VCYQaoSbhxbNeu2/dwhvlq7QrSoIyy0OB+HuGnTFwnjVsxjO080x1JlblQw7RpoYUMmeWC1M8A06Ce+VtOmEMc0/Is+JBE4vxJxXmtxyClBkoopJLZopJsW1dSUVJkl5s7FnFXXteVTphI693GQz5qLVlljpukcYqbwRIHy8+25sy9LTWYJseo6TbLT4kho+0Z3c+sTHg/4+YHrjW4v/WJyXV8H7nfDkripLxeByZxzIPtx7K6Lhxr7Fffz/fBVFBfwIVCoTADrIKlyPstfMkIEbEuIi6OiM91/z8qIi6IiC0R8cmIWL9QGoVCobBi8CTcYv5mhEW/gCW9SdLl+P9PJb2ntfZYSbdJOm2aBSsUCoVlY8ci/2aERUkQEbFR0q9J+hNJb46IkPR8Sa/tLjlL0jskvX936Ryg3mG1tT7qlpk5zbokTuo1pXnEUV8ybk/ipHzV253ZhThPLZS644OS899NrmVZuCrOGNpzzhpd5mRd6n/Aabp1dXItz9MMjFqfTZOGnPVkztH5AeE60rE3Tafcx9mKLKZPk7/M+TzLn2m4ND1jmW3mxHvG9rxDONNwsxWQdyZxBOcD2Ifeop3zDSx3ppvS4ftNE9dN5pWVn3qz681xwXkC1tXtzjE8h7DLSAc+XEmXme9lqwKvT65bMtaQGdqfS/pj9e/AoyTd3lrb3v2/VePjaCci4nRJp0vjD12hUCjsVayCF/CCEkREvFTSja21i5aSQWvtzNbaptbapmy7mkKhUNgr8CTcYv5mhMV8AT9L0ssi4iUaMZFDJb1X0uERsX/3FbxR4653U+xQbwZkWk5HLGwH0xNSWVIt/3KQMpLSmEoNOV3J9p7KfL1KPX2aQxzlEJeb9/Br3xIT65q2RUp9AAAgAElEQVSZ8wz5G3b6NDHKfK1SysruZ1ty9Rl/hV0uygKZaRDryrZwG16WxLGMrCvrYipLH7+Zs6BsrEh9XYb257PpFyUOUjeW+9HJtZRTvMJtDnEsl+k024fSjscj8yRVd1tQlshWyvEeSnpud0oJmbMfti+lnewZ4P2Z+RxNLTNHWpQY2C4em5lUsSzs43bAC34Bt9be2lrb2Fqbk/RqSV9qrb1Oo5W1r+wuO1XSZ/daKQuFQmFPMWUriIg4JSKu6Cy/zhi45lURcVlEXBoRf71QmsuxA36LpE9ExLskXSzpg8tIq1AoFKaPKWnAEbFO0vsk/YpGc14XRsQ5rbXLcM0Jkt4q6Vmttdsi4iELpbtHL+DW2pclfbkLXy3p6Xty/w711N+0iDSFs7F2JEJKQglirjsObT9uCk3KRn/DlC48s5xZLkg9bWJfkqLfmZwnlfNs9ysR9xWEN3bHoW1+3AZcXUY5wqDGzpVmble2JekjHff4PsolbAtLG0Orx9wWz0dctoLxFxFH2rypOw45NnI8t1qfR9jttl8SJ/XjYS4pkzQul1gaoZxDf7X2s8t+o5zicUXph7CcsglxlMms6XGsZtIVZQH2cebkaSPClmn4DLIsrPd8d2Rf81qnwbKw3I4fsqIwpipBTHcp8tMlbenee4qIT0h6ucYfpd+V9L7W2m2S1Fq7caFE98QOuFAoFFYXFi9BHB0Rm/F3+kRKx2n8uyCz/HqcpMdFxDci4tsRccpCxaulyIVCYW1iz5Yi39xa27TwZbvF/hoRxudpRDa+GhFPaq0NLUdY+RewKYxpLduHs7X+aeEnOq/1dz9ny2mGYdpKSkeJglTHVGmIrZiqZTPYjH8h4uhX1lSOW8/8DsKuC+lltqiEVP01CGe0m7LBPcn5X0b4DoQtc3AkZtIInfmQYrtduWDgsOQ8JY5TEbb1SmpUrnzLIsoG3qaHsgWtOCzTcCzQioJ1tfUD+4L5+j7KDhuS87yH7XbgxFEa337I9aLcQ76bOZHKFsowfz4jHu+U/viJR1/bHjt8Brlow+3G8rFdXQbWldYlHheZQ64lY7p2wNdqfKhlll9bJV3QWrtP0j9HxA81ehQvHEq0JIhCobB2Mb2lyBdKOqHzgbNeI4uwcyau+e8aff0qIo7WSJLY7W9KSRCFQmFtYopfwK217RHxRkmf14jIf6i1dmlEvFPS5tbaOd25X42Iy7qc/3Vr7ZbhVOsFXCgU1iqmvBS5tXaupHMn4t6OcJP05u5vUVjRF3BTr/N6dQ+12HmErR9Rn6SuadOuIf3L+tidSZw0rkFa16TZDXmD0+X9XJ3kOlH/yrQ4Mh3qYy9NrqXW5jb6xYHz1ghZJ5qsWX+jcxRuVc42ypaLs919X6ZRS7njbs5AuL8optG0y+3CFWfUydzGv4449tsXuyPrkenJ1Dp5P1foWdOnGRvbcL47Dq0q9BigBk0nTNm28DQvdF35DuHz4vijkjip7wvq8ZnDqcW0ha/hnnFZWqxr5qQpm+eRepGU7TsV7OMr4eoLuFAorE2sAofs9QIuFAprE6vAG9qKvoBDPS3KqCxNgGxORBMamq3MdcdtSZzU00amT6pHKmQqRRMh0i+X6/iB86b7pKI0TbI0QKr3iqOTDKBb3A8uus4cnZUl13NhKPdTA0kcDT+GtkfUI6wBsLBomJ/L5nSpN7iRaJtGncjaAx8M6kzOn7yf+Xed1VDmAO//LXd4xsWR19MQRT3oZHacv54yb0CS5rp63426bjgY19q+kYONg8wDD+V/5z/34f0mLpPy1ZY0U7s6Oc/sKX053WwoSeMykaUJPo+UQ9xU3xs4n3ULn0eXYciJ0pJRL+BCoVCYAWpX5EKhUJgh6gu4x37adSttUqZsZp3OP9iWvo/sklTKbJwaPNklWbPZ+BUD12Y+bLm6y/EsC6nYXHLP2PIvT1OjsuuyqXXGZXuCZ3uiSz2X/HxSKGl8ajszH2FjmFYPeX+29DG0PCzb456mA5YjWKbknuB5Ljt0vpRAMofHQ054n4PwT7sjBxG/qLr23kDzk4N2PT92D+vifsdDcM9f9mFLC3O4hc+DVSJ2Fa0I/AxwWFCCyOg+z9N51eHJeZYls265dyCcxTndoW3BloSahCsUCoUZoSbhCoVCYYYoDbjHOvVs1PSJ7JTyhCf8SZ9IeQ7VrqBzEKfLH0DKDpQrnC/94pKhegHE15I8pZ4q0v/pScl1NCwYy8wW6eRftBxwI5BfkuqaS/J+Wuebgv8LxLEBaV3htEjdOF3tsgw5yXUn0bkyy+J6s7Eyp868hysJLBfQ4TE9Mln7YV2zfa/mBvInXEYOUra75QSWj6tW3BaZdib1AxKWEU+EBOExOOTz2tWeRxzVFFeb3cPsLVFw0cqQH+jMtzCNV1xVyniZzEcFhhKEh9hUFYP6Ai4UCoUZol7AhUKhMAOUGVqhUCjMCE3j8to+iBV9Ad+j3mLIe1NRXrs9CdMpC/Uv66k0e6Gsai1pyFkP9WZrx1zw9EyEvQKPeVErm0vuzxzJ8/4xM6ls0zAK3j9NztOGKNt8LDP3YgHY2NkGb0Obc7mxhjwT0VbPYLksDFJLpZjogUHBnR3vjmP7sPzWtmlyx46x+RpFfk4IsFyZx3Nq05d3R4qoFFmzvDgIL5u4buJ2dxflcjavxyWHAu8/aOI6aVyX9RjnPS9BmKaUk3M30vg0wIMmrpPGh+B8d+Sw4v1+RjJztWWhvoALhUJhBqhJuEKhUJgRSgMexwHq2bKpCikHZQFTpWuSOKlfVUZ2mEkY2Tbdk+mazXKVz2HJtWTdDJs+8Z5vImyTNJZfZyFsrsbBwsZwwSkhZL/s5HzZ/uVMn1SbdN55Da2Ec2XXJXFSr72g4bdd3IfXm3dmpmtS33DZRnS8j3H3J2FqV5RIPtwdORieNHCtuT3LwgH1vYnrJq89uzuSa1MPsPTBwQJYmRna68/dmvlblvohkDnN4bXs6nmEma7vY/X4DD1s4jppvNp+XqjGUPqwJR7LPxXUF3ChUCjMAKtAglhwU86IODAi/r+IuCQiLo2I/9DFPyoiLoiILRHxyW6jukKhUNg3YF8Qi/mbERbzBfwzSc9vrd0VEQdI+npE/L1G+x69p7X2iYj4S0mnSXr/7hK6X7s62+BKt2xin0yT9McT30O+Uk1/fpTESTlD5/3fQtgubsngOfE+1x2/gjj6m03c8erT8Ptqhr8F549BY5hCHgBrAeZvKshtxOm72HIPJ+u5kIztmrioHTv/ou7IVYe0CPH9J27Nzx/Q/XMQIqmcmKpeg/qPbfX+/fF8eI/U9+ud2N+dbWVQAbkCznw4Bn0faX+2KJD1uwcJeAXl7biAast3u/5k/Rh2H9wxcP6mJI5yg8tP2s96+9kakumOS+LnlcPPDp9X9otX3W1CHNvShjInIw7K1dKx2r+A2wh3df8e0P01Sc+X9Kku/ixJr9grJSwUCoWlwJNw09mWfq9gwRewJEXEuoj4rqQbJX1BI091t7fWtneXbFW+96Ei4vSI2BwRm7dnFxQKhcLewv2L/JsRFjUJ11q7X9JJEXG4pM9ofI55oXvPlHSmJB0e0TxbavpG+kmqblo9JCyb6Q0trrCss9AurlIuZ5B2zndHUiqW1RSPtD8rC2d9SQvdBpRFuPf1XHekzxdSZVO5zBWtJH22Ow758iEV9DoG0nbW+1PaFZl1CMtPOWZ+Ih9p3PDAv+LZbL7UfzEwT7alKfCQxYudwtCFMJ8/jgGPLUpLmT9dygqsi2n3DQPnbdTCMcz8bVXDXYz4vHiMsq3mEfa44+5T2c7Z2S5T0vgYc7p0l8x83e7cjZuSmNcScaxRHvSzM+awarlYa2ZorbXbI+J8Sc+QdHhE7N99BW/UsIvuQqFQWHmsgqXIi7GCOKb78lVEPFjSr2i0CPN8Sa/sLjtV/YdWoVAo7BvYxzXgxXwBHyvprIhYp9EL++zW2uci4jJJn4iId2k0YfnBvVjOQqFQ2DOsAjvgBV/ArbV/kvTkJP5qSU/fk8z2U6/nWT+ifkZ9y/rY0Lb1/nSnuVi2jdgPBtJ/EcLWC6mRUldM/GaPmdtYa+MPaaaf0dznSwjPTZRDGh831mO/n8RJvd7LtqRJ23O74ycRx0V1rJe1OmqF1OqsHQ/t+v7x7si+oMmaNUD6tLkFYdeRWivHgPtzDnGstxea0dyKbWXtl6sW2S/Um50WddlsAR7Psy3/vjtyboDjwvdxcRxNJV0vjkXOA7gvOG7Yrm5rOknfiLDjWWemRT3XWGh/ucz3vdTXhXGZ+eC+7JA9Ik6R9F6NqvOB1tq7B677TY2mS57WWtu8uzQXZQVRKBQKqxJTkiA6BeB9kl6s0bzmayLixOS6QyS9SdIFiylevYALhcLahL+Ap2OG9nRJW1prV7fWtkn6hKSXJ9f9R0l/qkV61lxRXxD3qZceTDkyV7RST29orkQXr/7pIb0jPTI9IyWkCQxpsUFaTjMtU0VKAKSC2U7u/GWztHEh4mh6lG0vTnpnqkiqzLpYWhmSFezr5TGI4+RwRrFJD9lHppisPymu243lJy11v9NkhmZa3ueMzwQliswvLWUBtwHzzMB7aOpIuu/FfNmWdFLf7pQ7KP14bFIWYFubbmdylZSbH2bmd0MSivuCq884hi13sK3Y7oz3eGH5+YbZLznPfqNZZwaXO3MnvWTs2bb0R0cE5YIzOxNa4ziND7ut6hc7SpIi4imSHtFa+38j4l8vJtNyxlMoFNYuFq8B39xa27TwZTkiYj9Jfybpt/fkvnoBFwqFtYnpLsS4VuMkc3LtwyEazVt+OSKkkYfOcyLiZbubiFvxbelNTc0MSAVJU26duE7qZ/OlfnXS0OovU9XMaY00blHh1Vekd6RXXvaXWR5IPW3K/BlL/cw9089WNx2fxEk9xWSalDAMrmhiWTwGSQlJ+7PZatY/s/jgPbw225787uRaWpxwxZQp7FWII4X3E3BlEif10gz7hxKD6TzLx/LfmVxL8Fq3IfuVcoDbgmXltf4441M8h7DHNsdCtt6ftJ3Sl9uNjpeIzMcvw0w3G4OUwQ6ZOErjMpev5TPO8eq2oBXIVDA9K4gLJZ0QEY/SqMteLem1Ptlau0PS0f4/Ir4s6Y/KCqJQKDwwMcVJuG7F7xs1moq6XKP1EJdGxDsj4mVLLWJJEIVCYe1iiqvcWmvnatzNiVprbx+49nmLSXNFX8D7qaeD9nFKCSGz2yB9I+1+YXfkrDVppakS6emQ8b+pEh20cCGBKTSdj7BfLR0M+SY2LRsaC6aCvJ9ldfnYVmwXx3NzX6Zlqkx/wDTIJ+3zLH7m71jq221o99uMUpF2emad93BRhOUalontZrrOuhLOnxIJJZRsjPHabIPkbPsqnh9yKOV6s01oWeA6UpZgH3s8cmaIebkvKGdlFitPQdy9SZiyAWWFzJEVxxWfrUnrJmncY5fbkO3Dcpunc1wuG3tmBTET1BdwoVBYm1gLS5ELhUJh1aJewIVCoTADrDV/wNOA9TB796Gpy+3aFdSRqPd6oTVNt/hj53anZkZkuiR1R+rBNkdiX1JLyxxaZ9vWU9fN9kFjmtSbrUtmzopYPp6nbmkNkRr5kIMZp8G2oF5uvZCmU9T11iVxbJ9JM8TJ/L20iOXPHDJxLFBjdbpzC5Qv23JdGl6BZ7DdnC411u8h7P4c2ifN7ULdFFvZ7TS7ZJ6Z2eU1A+dtOja2Jx/C892RK/FYF97n1X60qcragmAf2bFPZtIojc8DTBX1BVwoFAozQE3CFQqFwuywj38Ar+wLeJt6umSqO7T67GFJHOmVTVhIaYgDJ46TyFaNkcIfmZxnXjQDs2MglpU/vNledNn+d6wfTYOcLmUJSiiWcbhyi+ZUrtdjk7jJfJ0GJQjW26uqSBlZP8s0TJN9bPND3v/w5DxXx5Fiu93oBzAzz8tkCalvQ5aJpmXzCD+zO2YOdKR+bA2ZSvrhz1aMTeZrcNxlfqa5gtF9NeQ72WOc7cPVac/vjvSbyHbjy8sOnTiu2BaWO/hcsY88RlkW1sv3T/ODdRUYQdQXcKFQWLvYx+fg6gVcKBTWJuoLeAL7qadopkqkn1zFM98dT0Ic6Y2pMGfjSY82TlwnjdPebFv2ewbOZ1YSpFem6CwLaadpM+UQ3m+Ky8Hyo+Q8Z6hJdW0RQUrLtnJbs360yCDt87bjpNUst9tgHnFzCJtKfhdxdIZj6Yi0mRKH5Qa2z08RtvTyTcTRz7HTYpkoDbl8pNqUGOgkyHIB25Uyj/NiWpR2XNc7EMd6uY1p0ULLA1oRTJaJ5eI9lCj8PGW+raV+W6xHDJxnH7le7CvKaB777CvKHR4Dme9mqffPzdVx00B9ARcKhcIMsEP7/K709QIuFAprF/UFDBygngKaapIeklZ7K6L5ifsNUx5SFtKnbLad9Ix+W03rfjJw3rSbsgBp6WUT10njO8raeJ30lNe63kM+bi0hME861snqmm3PxO2dSJvZ7pYuaGXBNvbiBDpqyXwj//rAefd3tjhD6hdicEEDpRe3wfMRx5l/S0+UEtiufiAp9ww5MXIbcSxwIYLHDfelyRZ4MC9aoriP2ZfcxdtjiGWiBOF6zSGOFiOuV7YTs9T3O2UBpk8JxHkwfY4LS1qUs/jy8xjIxhrTWmgrqT1BacCFQqEwQ+zrL+AFHbJHxCMi4vyIuCwiLo2IN3XxR0bEFyLiyu54xN4vbqFQKCwOdgUxhV3p9xoWsyPGdkl/2Fo7UdLJkt4QESdKOkPSea21EySd1/1fKBQK+wS8Enkxf7PCghJEa+06Sdd14Tsj4nKNtqZ6uaTndZedJenLkt6yu7TuU6+DWvejPkX9y7rbHOKoX9mchzoVHfeYevAXhuHMROZpiOOqMzspzxyHS73uyLpcjLD1Xupnv4PwrUncPMKPmLhOGjfZo4ZqUO91XR8Me6TbUNgjEH9jF5+tcmL82J5ruH/bPbuWj2ZYrgvNlWjm9uCuk76HzxI66fYY4N5o7FdrzDR9Y19bQ2b/Um8P1GW+qwv1+ldqV2T79xHUOucQ9tilLnw+wh4v84ijyZ3rSvNHzj24CalLM+w+omkcxzDzcntxzoZzIh4P/Jqknuznhf3CeruNpv0y3NcliD3SgCNiTiNHZhdIemj3cpZG/ZPtYaiIOF3S6dLwsuBCoVCYNtbUJFxEHCzpbyX9QWvtp93Wy5Kk1lqLiJbd11o7U9KZknTowDWFQqGwN7AmzNAi4gCNXr4fa619uou+ISKOba1dFxHHSrpxoXR2qKeQNnMiLSd9Oyk5T/pk2k0TIvpvdcMzTZozfRFhUtDJ9KWeapGKZo5UCF5r+kb6dej/jbDtiUARHkN7nY5jPyzb81zKnbFCr3iweSPo9RHkzSjYQ5Kk+Bmx/sBd42gHtr7Laz0lDjbGFUlcsq/9k1gn8NIjHrprHLWpBzv+aMTRuXOnUz2S6Q84jZ67O4nkEsLu/EOSvpK0sz8fQ9sq9NuTEju3b/5+H7Y0QObIYrvaQz6zbYr3kyRO6p8XdsX3EeYz4DJwCLIsHm4LOf5hWRieS+7J/GzvCVbDF/BirCBC0gclXd5a+zOcOkfSqV34VEmfnX7xCoVCYWmY4q70ew2L+QJ+lqT/RdL3IsJL/P+NpHdLOjsiTtPoI/FVe6eIhUKhsOdYBf7YF2UF8XVJMXD6BXuS2Xr1dP+q7kgmTXpj9kcJgvRpvjuenMRJPWUii/wOwkzLs8ikXJwNNtUiZbo3Oc+6kFo4rzHnKm+mIGI7i7k+6rZ/6sNH2KaBa75w7c455oMRR9uBu0aH69ACxw65Pela7qco7aGvwXlvbESyyrLc3B0pDs0jfFISR5sEz+OzhbcnYRLzZyFsvYFDewvCv9Yd/xZxBw6EXQbaFnCu+bzuyLVw30DYZYAesrN9cH5br94dAwnCtJ6rErNnZGg1qccr78m2/eJWVUyLKwidBi0baJVjueQ7A+dvnbhuMi9/2dESaRpYExpwoVAorDasBg24XsCFQmHNol7AABdiGKQ0pPimRZyhZWN6MpoG/dk2PnTAQwJ/VRJPCYHShdMlOSVtOyY5T9K7fuK6EX5Du8UR4KI7S8vlH6S93j+XsgAliI62H8vlCySAlC46On/oXYjjMHlqd+RGNpQbvE81afs3kvO8n+UyXR8amm75ecQ9F+HXdUfKFqyr2+jPBs5zlFjOoGxAN0TuA9blTQhbruBSCdarK+P6fsnDTViO4y2RWDo6KTohOZ8tSiH4DGX+iml5kG0rxfs5ni1Y0TKCkp4tNrh4g2lZepi2M55pShARcYqk92r0qvhAa+3dE+ffrNF6qu0aver+t9baj3ZJCFjMUuRCoVBYlZiWFURErJP0Pkkv1kjKfk3nkoG4WNKm1trPS/qUpP+0ULr1Ai4UCmsSU/YF8XRJW1prV7fWtkn6hEbuGPr8Wju/teb5+29rnIamqBdwoVBYk5iyHfBxGjfK2qpx7WwSp0n6+4USXfE94awr2bH1s3E+04zmEEcTF+tf1JSpX1mP5SohrrJhXlYluWCKWpZNe2jCw18uOunJ7rcCOP5L+18QnktymEc4+yG9IomjudN5CFuPTUygJI2ZgW3/THeaZmpUt60NUyu9CGFrr9yNjxrzScl51s+tTfWey89cb2rULF9mGkbFv2uj9tU+KrhGkXq2+4NtxbreNXGU8vVIHIVot592ZcDkxb/Ald9KUvpZktKVSZzU9+qQw3nrrTRdy0w9mQdXjVLcdBWYVuacnrryPcn5bFHncrAHGvDREUF/+2d2bhT2GBHxLyVt0vjkRIqygigUCmsSe2iGdnNrbdNuzl+r8d+fjRqfXZUkRcQLJb1N0nNbaz+bPD+JegEXCoU1iymaoV0o6YSIeJRGL95XS3otL4iIJ0v6r5JOaa0t6BtHWuEX8P3qV8R43fLQPmsmwJQVuAbJ+2jRFIZyRObMhyY82bbw/HmjRGHHPZQQSCp9P/2b0iDMJm00YNK7YHBzdBce+r3c9sPRkXyKhbEN0PGIYwXWd5u4szFoZ0fe6Ia5BS3PfG3bdOSP87K4E8k/74ex4FEd7Z7H+R3Iy98g7HiW1YsCaXN40Mf7sHntIX+nFOblY0ssIU7th3pZseHAOgzruxyk2pFtdrcFqxo5iN3ueEvwefClXFH2ZITd1HwGeP/DJ66TxrvKXTSHuO8iTLnCY5dlodxxSxLHYXN8EkdYepgfOL8UTHMpcmtte0S8UdLnNWrSD7XWLo2Id0ra3Fo7R9L/qZHe9jedt8gft9Zetrt06wu4UCisSUx7JVxr7VxJ507EvR3hF+5pmvUCLhQKaxblCwKgFYS3XiFrJmv1vHdmYSD1c+icNSUVu3PiKI07+mBepk20K6CcQIZukAE7X64o4myvy0gme/2/68MmwJkbGKmfxWZdyMpNC6k6kHZmLnw5Az2HRvwf9+2aF+G0WH/SvPnuSAcytGd4WNcwDQUIGkm8f3S4BYkehUGyrePYrH9WVtpNZF9BQ1u9sw8yNYVt7HY/5H193HoYdPy0G6Rsa45n388xfCHCc91xHnHZuGBbcP2h5QiunzwJYdeFM0lsN8JpsfxsV9clkxGlfuwzjuPCeD7CHx0oy2JRviAKhUJhRpj2UuS9gXoBFwqFNYv6AgZCPd3yxDEnjSkh+JeLlIhUzZSIVJ+/dpYVuPiCIO30xHjmXGSyDEa2lRK92pIq2rCA6XwAYbNt5v8ghE3VuOUSrTBM79gWpNhO956B808E3TfdHuoXU3TWL/OdzPal9LKju5j9chTXMXQgrT4QU/uZnEPrFZ/nbD1lBdebDyYXH7APnQZn9ucRtmXA2PZUWF/y9e7IfsncFg3583W+v4I4tovbnf1DJ1CWGCg7ZHIKLXaegXAmiQ2V1W3Be9jGlvfYL+w3j23KJcvFmnDIXigUCqsRpQEXCoXCDFEv4EKhUJgBahJuAjvU61bWfOhQnQ5urCFSc6K5jM1ZhvZhM+iuiIuQ6NjH7rDp/D0zU6J+Rl3PeilNbHi/NVyuUuJKPOt61L+or/lXnBo4rcGtFbKtqLXZNInub6gbvhNha3k0qaPpkfVS6r50dbNu4jppXE92W7J81Nv9wDwecXSS5PSpIWe7vg+Zqbne7OtPI0y3PK43tXe6Xne+HHe8f7/kPPv4S0lZnomwTdK4Om0OYWvI1H2pvc93R2rIf42wTQWp23KMZKD2zj7K9k3kM+KyMi+2q/uFY5za9FJRX8CFQqEwA9QXcKFQKMwITcNWUPsKVvQFfIB2Xb0z5D/UdJpmJIcm57l6jVTPv3z0NEsqSr83ljZIr0iFbp84Mn1J8gbzXMVEkzOb+5CSzSdlIaWjj1fTSpaf6Wf733GF4cXdkaZtQ9TMFJG0lnTe/ZGtJGQZKfFQGrGc8MQkTurrekgSJ/UUl3EcQ6a17B/SXoPjhqZn2Qb1Q9vCz3dHtvsvJ2Vlmpl0xbakZGZpgtIW62UZh32ZOcjhWDoZYfcR5TDKHWzj53RH9lVmZsYxTAmBMksGX7vQdXuK+gIuFAqFGWA1mKEtuCVRRHwoIm6MiO8j7siI+EJEXNkdj9i7xSwUCoU9w5S3JNorWMwX8Eck/YXGfWOcIem81tq7I+KM7v+3LJTQ/eqtHkxlT5g4b5hekXZTjrBEQKrN1Wl2psPZds7ck3Zlv0KkT86Dq5hIqy1dkJ5SIjDFpKxBucMWB6SnhGkUy896HTlxlMYtD5wXVx7xfloseJaf7UO67j4gPc36iPXL5ARaPpA2m66zrzkujpy4Thq3wnC95hDHcTGZjjQ+Bukz2nmwrKyr+5OOm9hWbm/O9nMMeDUo8yQF97OyZeD8vUkc22KuO7L+7HffR9nhJQhTbrCcwnGfbXHPfjsmuZbSHyUxl4VjdBrY1yWIBb+AW2tf1Xi/SaPdQKXnSj4AAA6mSURBVM/qwmdJesWUy1UoFArLwpR3Rd4rWKoG/NDW2nVd+HqNm9iOISJOl3S6lE+GFAqFwt7AatCAlz0J11prEdF2c/5MSWdK0oaI5k9pUw4uriBdOGbiKI3PFjsdzhATXvTA+x+enOc1pFeZ31PSAM722nidlIoLTPwLS7rBBSQv7o6kigybttKkhrR2Mh9pYQc6lAhYbi8EIC1mXU2xWX7mle1qyx9e9zHvIW13XpRbKBfYYuGOgfOWibjggXKQ+5ULBjju5hB2XYasSyzXsC5sF7cxXwJMy4temD8tMlyWTHqTepmIchPTslxCiYVj0HumZxKUNP68ON8h/9yuK9NiW8x3R8pNhJ1LXTJwfqnY11/AC0oQA7ghIo6VpO64qA3oCoVCYaXghRiL+ZsVlvoCPkfSqV34VEmfnU5xCoVCYXpY9VYQEfFxSc+TdHREbJX07yW9W9LZEXGaRmz+VcMpFAqFwspjNWjA0dqgfDt1HBLR7BzaeirNnahR2iSLWi31p2xvMprrZE5XqDVSN3S+2X5gUr+SiCvCSFus5dFEKXPOPmRyZ13wEQPns/Kxrea6I3XhbHUYdU9qcayrdWauyGJZFlrBmM0os6zuF+bJ9J0u2/qm5DzT5Ayw25ha5pDTfoPtSkro9qKuyRV+7i+OS6bvdqeezXo/KInjuLGTKJqJcQy5XXg/x5Drzfw5RjKTv6H98TxeuKqOZXW/8LnKHPOwfNTDPQY4p/NR6aLW2iYtEUdH7H5PeODDy8xrqaiVcIVCYc1iX7cDrhdwoVBYk1gNEsSKvoA3qDdz+kp3pLkVqVy2eou006Y3v4A43p9tVU56RIprMzJSWVIxx9PXK2lt5qyH0ofNoJgnKfzPkvOZ6RHj5hD2Si06uMlWANIci6Zn2V58pP2sq/Nlv2V75tFkjhQ5k1NosucH5qYkTuolApaZKxQpMxmPQdgUmnViWbIVhrckcSwjzcBYbztpotzDerk/WX62lVe1UaabQ9gSAMtEicQmc8yTFD/zzcxnLJOTWBdKDB77fAYzaYj9wz50f2TP7XJQL+BCoVCYAcofcKFQKMwQ9QUM3K7eYNhUaGzLcoQfn5xnY3rlDJ3mcMWW7xuaoacEYLkgc7Qi9X6I7x44z1VZBi0yLF2w/JyZNq1mmqRnpn2kb1zplTkeIpX0/UNbhq9LwrQGyHzkkoqSzrtemZWG1K90Glq153wpPWVWGkOyge8j1aaE4DHCccF+Z77Oi+OS0pTLNTRuXC/mdUhynmOc0o6lBdafcofjabHCdsmcRBGWjth+zIvj3Wmw/Psl5ynTsV0sk7EtWW/LJazfcrFD0/XzEBGnSHqvRs30gdbauyfOP0gjp2VP1WjY/VZrbX53aS51IUahUCjs85jWQoyIWCfpfRp5DjhR0msi4sSJy06TdFtr7bGS3iPpTxdKt17AhUJhTWLKS5GfLmlLa+3q1to2SZ/QyCskQS+Rn5L0goiI3SW6ohLEftrV+JuUh/TmsiSO9MpOfGgFQYrueFKaoYUWpsv065pRPc4206+pyzi0kMJpcdb4hQgfn5znzLgd4zwbcaRWLhdlDVLwL3RHygaZxCH1dPlpyuHztPLIHOtwNpy0NPNhS4N8p/U1xLGPKR0Z2RgifaZEYmloaAEQafFckj4XH5his3yZQylaeVCuspUAFxvRosRyxJCMZrmD5Wf7uCzsC45bp897mNbtybUc15mjLOZFi5DMEijbaortPw3sgQZ8dERsxv9ndo7EjOM0rhBtVe/SeZdrWmvbI+IOjap+81CmNQlXKBTWJPbQDvjmWglXKBQKU4Idsk8J12qc5G7UuDddXrM1IvbXiBBknmN3ojTgQqGwZjFFb2gXSjohIh4VEeslvVojr5AEvUS+UtKX2gLOdlb0C3i9er3Tuh91V5qUWbejzkRHIDYNo1ZLZ9nWkKn/0SyG+pb1Vupj1FOtHVOrpFbln0WahmUmX9RlX4+w9a8XIY66oOOHnO0sNIA8Ilhnmktl28oPmanZjIsaLoWxzJE++82rw6gFUne0LkiNlnllnxNsC6d7EuLYV86LX0ZDe/FZe6cDHI4L9/uFiKN27vihFYoG9f5vInx3cp5wf7KvWC/3Cz/T2K5uF/Z/ptsyLfYrNdy57jjksMrjim1NPdzxLB/nRJaCaS7E6DTdN0r6vEaPxIdaa5dGxDslbW6tnSPpg5L+KiK2aNTVr14o3ZIgCoXCmsU0F2K01s6VdO5E3NsRvlfS/7wnadYLuFAorEmshqXIK+oP+LCI9qwunO2TthVhm+gMORIxLSaloYTgeNJjUi3+MppqkaLT5MxlzfaRk3rpgZSMMB0m1f4QbZdcWOoplPvnuyM1FPJSp8UNtagx3JfE0Z6KXNOFZAORK1oP4P3ZHvXMaw5hNyz5JZcwug6ZQ2Op13b4ZJE3u13Y2Nle7EPOobNOvCeJk3r7sicgju1yYBLHfnO5UP6T7urDHq9sSnaLk+JY5BDIVliyKz3EKBsQbAo399BegW6ioY13/WxlcpPU14VD/HPL9NG7IaI9YeHLJEnfKX/AhUKhMD1M2Qpir6BewIVCYU2i/AFP4D71zNNUhbOepC9mxdkqLamXFng+m9knu87om9Qz/4VW6XA2+zKETZvmEEfpxOUe+zV+BsI2DeBNzOD07kgNhI1l577PRVw2zc/72VjUdmzqMY84NnKmDZEDe9kbtR/KLeagdFw7h/CvdEdO3ZOXOi/KAkzfnUVtim3h/ClLsLMpA7nDqY1lHJrp09GyNTGWJZM7UJZNb+vDVkmosNAiw91J2YGygMfzkFrj5yFTgKTxIfb57ji06s/XUlLMVtXRooXWNe62eU0P9QIuFAqFGWJfn4SrF3ChUFiTqC/gCRyonkKZYZJd8tfKDHbIGY/vew7i5hG+feI6aZx1Mq27k7jsPCfOWVbvmsu86JTEtI4LRfRKhM3FqItQFsgcu2ZWDkMrFjKvMUPW986DEslNSZhl4VSz09qIuGyFzTMRl1kesPy81h1DieMrCLuO1J5YP/Niejai3EOObz7NVR2UEFyGITMDW0kMbUHtQYr6Z1siUeGg8YjH090D5z0Gh7biclEuSeImYa8zbFY6GbJFRfYMS30dDkVcpuawqaiYLRX1BVwoFAozQNP4b+++iGX5goiIUyLiiojYEhFnTKtQhUKhsFxM2R/wXsGSX8CL9BBfKBQKM8MUnfHsFSxHgtjpIV6SIsIe4i8buuHRT53T32x+x0TW9FVM1cfGMVSKWNw3dMf3Iu5ohL2k6OCB84RFRPpX/i7Cr+uOf4u4TE2jayEqvv+YpJ+5FudQ+DzC3vWL7oCoxmXLv1jvb3THxydx0rhjf2/izvwpTjMPg/W2sHlpH/WbG5JrWb7rdw2/guepyHsMQGT+XbpBmuuO7D8qi0/tjhf0Ub/E+m1H2HW9CnEQ0p3U2AbsqPfOstKAkmNg10f/mN/vXfV7zoHSPuV4W/JRtqbues/EURp3jp+B0j57xTI3W4J70WXWfXxCLOPTupAj2HkN7V+3FKyGSbjlSBCZh/jjJi+KiNMjYnNEbL7ppjsnTxcKhcJew5qVIBaL1tqZrbVNrbVNxxwz5C2hUCgUpgsvRV7M36ywZGc8EfEMSe9orb2o+/+tktRa+z92c89NGjGTwT2SVjGO1tqr11qsk7Q267UW63R8a23JqkRE/IOGdcdJ3NxaO2WpeS0Vy3kB7y/ph5JeoJG0c6Gk17bWLl3gvs2z8Dq0t7EW67UW6yStzXqtxTo9ELDkSbghD/FTK1mhUCiscSxrIUbmIb5QKBQKi8MsNuU8cwZ5rgTWYr3WYp2ktVmvtVinNY8V3RGjUCgUCj1qW/pCoVCYEeoFXCgUCjPCir6A14Lznoh4REScHxGXRcSlEfGmLv7IiPhCRFzZHY+YdVn3FBGxLiIujojPdf8/KiIu6PrrkxGx0ErWfQ4RcXhEfCoifhARl0fEM9ZIX/2rbvx9PyI+HhEHroX+eqBhxV7Aa8h5z3ZJf9haO1HSyZLe0NXjDEnntdZOkHRe9/9qw5skXY7//1TSe1prj5V0m6TTZlKq5eG9kv6htfYEjRw5XK5V3lcRcZyk35e0qbX2RI3MQF+ttdFfDyis5BfwTuc9rbVtkuy8Z1WhtXZda+07XfhOjR7o4zSqy1ndZWdJesVsSrg0RMRGSb8m6QPd/yHp+ZI+1V2yGut0mKRfkvRBSWqtbWut3a5V3lcd9pf04G5B1EGSrtMq768HIlbyBbwo5z2rCRExJ+nJGrnWemhr7bru1PUad5G1GvDnkv5YvW+SoyTd3lqze7DV2F+P0siB2Ic7aeUDEbFBq7yvWmvXSvq/JP1YoxfvHZIu0urvrwccahJuiYiIgzXyT/kHrTXuzqI2su1bNfZ9EfFSSTe21i6adVmmjP012nP6/a21J2vkh2RMblhtfSVJnWb9co1+YB6ukW/TFfdjUFg+VvIFfK36raOkkUPXaweu3acREQdo9PL9WGvt0130DRFxbHf+WEk3zqp8S8CzJL0sIuY1koaer5F2enhHcaXV2V9bJW1trdn576c0eiGv5r6SpBdK+ufW2k2ttfskfVqjPlzt/fWAw0q+gC+UdEI3U7teo0mDc1Yw/6mg00Y/KOny1tqf4dQ5kk7twqdK+uxKl22paK29tbW2sbU2p1G/fKm19jpJ56vfPnRV1UmSWmvXS7omIuyJ/gUabRiwavuqw48lnRwRB3Xj0fVa1f31QMSKroSLiJdopDXaec+frFjmU0JEPFvS1yR9T71e+m800oHPlvRIjbb2eFVr7dY0kX0YEfE8SX/UWntpRDxaoy/iIyVdLOlfttZ+Nsvy7Ski4iSNJhbXS7pa0us1+vBY1X0VEf9B0m9pZJVzsaTf0UjzXdX99UBDLUUuFAqFGaEm4QqFQmFGqBdwoVAozAj1Ai4UCoUZoV7AhUKhMCPUC7hQKBRmhHoBFwqFwoxQL+BCoVCYEf5/4/DxjQAU1VMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWYAAAEICAYAAABs2F48AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztnXu0JdVZ4H9fXxpaSNMdHvLoRppoaySoIUMgTpwRTaIEs0RHh0BGQxIiOgvGRKMjybgiQ1ZmyIwhgyODdgADrhjC5DHplekRYx4TdRKkSYwJTZi0PELzbp4tTPp1v/mjanO+e/rbZ1edc+69dc75fmvdVfvuvWtXnXOqvtr17e8hqkoQBEHQHVYs9wkEQRAECwnBHARB0DFCMAdBEHSMEMxBEAQdIwRzEARBxwjBHARB0DFCMAdBEHSMEMzBTCIibxKRv17u8wgCjxDMwdCIyEHLdZylOnYQLAchmINWiMi9IvI7IvL3wLMi8j0i8nEReUxE7hGRXzd950TkXSLyDyKyS0RuF5ETRGSDiKgVriLyBRF5a11+k4j8jYh8QEQeBy7z6uq+bxGRO0XkSRG5RURONGOqiPyaiHxLRJ4Skaul4geBPwJ+VET+UUSeWppvLwiaEYI5GIbzgZ8BjgA+CXwNWAe8Cni7iPx03e83675nA4cDbwGea3iMM4C7gWOA93p1InIO8C7gXwBHA38FfKRvnNcBLwd+GDgX+GlVvRP4NeBLqvoCVV3b+JMHwRIQgjkYhj9Q1fuBU4CjVfVyVd2jqncDHwTOq/u9FfhdVb1LK76mqo83PMaDqvpfVXWfqv6/TN2vAf9RVe9U1X3AfwBeamfNwBWq+pSqfhv4PPDSET97ECw6oacLhuH+ensicHyfKmCOauYKcALwDyMeY1DdicBVIvJ+UydUs/f76v8fNm3PAS8Y8nyCYMkIwRwMQwpJeD9wj6puzPS7H/he4Bt99c/W20OBZ+rysZljDKq7H3ivqn64eMblsYKgM4QqIxiFvwV21YuB31Uv9p0iIi+v268F3iMiG+tFtx8WkSNV9THgAeCX6n3eQiXA2/JHwDtF5CUAIrJGRP5lw30fAdaLyMFDHDcIFpUQzMHQqOp+qsW1lwL3ADuphPGausuVwM3AX1DNjK8Dvqtu+xXgt4HHgZcA/2eI438SeB9wk4g8QzUzf23D3T8H3AE8LCI72x47CBYTiUD5QRAE3SJmzEEQBB0jBHMQBEHHCMEcBEHQMUIwB0EQdIxO2DGvENH0hIilyCAIBjEPO1X16FHGOOuss3TnzmbGOLfffvstqnrWKMdrSzcEMz37qv0N95lvMX7TMYOgq5xtyqfV25NN3fGmnAJ/HGbqDjHlub5tP+l++Y6pm3fabZ+9pm6v02738e7H3P2cxrKv9j/R8+ocmp07d7J169ZGfUXkqFGP15aiYBaRE4AbqQLHKLBJVa8SkcuobFEfq7u+S1W31Pu8E7iQ6jf4dVW9ZdAxVgCr6vKh9XaVaV9pyrY+YT0EVjrt/ceC/IXkXTT2ArZ99zjtw2CPaacBR9bbI5w66D3MVpu6w005fZe5GzR9lwdn2g92+pZ0X/YGW5GpH4TdZ6/Tbr+ruUz9oGPm9i/9lva6anpeubGSsCp9P7b9EVNON9yDpu7rTl8blGS3KSe3y13OOdly7h7ZY8rzTrtlf9+2n6bXxfgnVwrsG/uo46LJjHkf8A5V/YqIrAZuF5HP1G0fUNXft51F5GSqIDYvoXqQ/6WIfH/tjOAyT+9iSReFvajtTZEu1vlM+1xfP1goYDzBkhPm3o21x6nLCfY5p87iXZTbTfmuwvjzTl3pAi7dCPF2EcwGysLHUbcoCmZVfQh4qC7vEpE7qYLE5DgHuElVdwP3iMh24HTgS7kdVpoB04xxjWm3M0LPf9YK1jRLzD3BU31uNuPNBnLt3hhWsKVjlWbx9lx3OWVvNmPLJcHdZjbTZpYbQjyYXCZ/xvw8IrIBOBW4FXglcImIvBHYSjWrfpJKxn7Z7LYDR5CLyEXARVAJg/TalcKUlV5ZczqvEm36podA7lxWOu0eVjB75517JW86Iw59exC0ZUoEs4i8APg48HZVfUZErgHeQ/UJ3wO8nyoQeiNUdROwCeBgEU3CK+lIPfWFJTcL9FQZlvmG7fYYuXNpqsNc4dTl2u2MOOkF7Szd0/t5wry/PuF97iaCPQR6MF1MgWAWkZVUQvnDqvoJAFV9xLR/EPh0/e8DVHF4E+vruiz7gSfq8hODOgZBEIyFCRfMIiJUUcHuVNUrTf1xtf4Z4OfpxdzdDPyZiFxJtfi3kSo8ZP4YlPWwicVYxW1jVdFmxpn6lsYvqTq6t6IdBNPABAtmKl3yLwNfF5G/q+veBZwvIi+levTcC/wqgKreISI3A9uoPvnFgywyqAfIqSaWgsXSUae+pc/W3bXhIJhW5lloRNgtmlhl/DXVpLafLQP2eS+9BJpBEAQdY8JVGUEQBNNJCOYgCIIOETPmIAhG5IdM+UX11rrql5ywLCXv1b1OnbcP+E5OuXET3mJ4znzVcwj7WKZvO0IwF1lB78JKFgqevS/4lgqe7XHO3rjk1DGqS7PFO9agfsuFPT8bi2SFU7860zeNYS1MvBgn3j6WnM24F3/Ca89ZuHiu/J4wyP1WbeJaJKyA9K4x217yKP0RU05C+kRT90Ib5CQFRznU1HlffEkaetGI+uu9FW7Td//8wm79Q3m7e8M/Z+rGI5jn6fKyeycE8zwLXZGnjeUWvCXs+T2b6ZN+n8cy7cEys9sph1NAge7OmCNQfhAEM0hSZTT5G4yIXC8ij4rINzLtIiJ/ICLbReTvReRlpTE7IZiF6k3L/q3M/M0V/oIgCMqMTzADHwIGBdJ/LZWj3Uaq+EDXlAbshCqj2wH4giCYPsa3+KeqX6wDvOU4B7hRVRX4sois7fOcPoBOCOYgCIKlpZVgPkpEbLqTTXUQtqasA+43/6eImyGYgyAIerR6T9+pqqeVu42PEMxBEMwgS2rH3DriZicW/4IgCJYWpTIUbfI3MpuBN9bWGa8Anh6kX4aYMQdBMJOMb8YsIh8BzqTSRe8Afo/a10lV/4gq4NvZVCk9nwPeXBozBHPHeLkpn1JvX2zqrLfXhnprM2sfY8qSfHatu54teynJc1lwU3lY39s08Sh5kFm3Os8dzu7jueMN46JnydlcehMney4lP2N7Luk7yOUsc/Z/8p5eeVu9vdt0tStLyQnI+pc8ZcrJWch605VySeZyYZZSvXU7FdrYrDLOL7QrcHGbMUMwd4zbMuWhiLQwQZAhXLI7SS7mQgkv1kIurkdqz8Vv8PAmnpMzAwmCSSGCGBWxQYzSW/Va076KwZTeCL2gKTlhZsdKQjIXWcujFCSnROlNPAiCcRCCuYgNYhTBcoIgWBpCMAcNsQt5xzp1dnEvvVXYuLy27+H11lvvAziy3tqIkYebsp29pzFsqMo528GL12rbvSDBw+iQcll7m8ZrzR3Tez0pvSqVXmlKWXRNuzpj2apvmfK99dYu+D1oyqXFv7ToZyMJemuy0734FzPmoAWPZcqdxN5h3c1rGQQOIZiDFhxvymn2e6yps+VjBtTZettuZ9eHp9nn0ZkOdqrtBV8/zJTTTNQuCNiy3c9rT+XcjNhbPMiZ9nl4M3pvOlZa0Mgd34vaX8rWkMNL22GnzHfV238wdcZ27smd1fY+0+zNnnOTgMfrrX3WPmPKNnZ6OtWc6V3JStJbxynNyO2xhiesMqaeNqavJR50yl8bYpxGpLvCO2gQTD0xYx6I0JvQeBMnzxzNCj1vslTKe+ZNcHJjWUbVeZXSXHkzi+nT7wXBchOqjCKKv+AQBEGwOIRgLmJnzE1ndDGbC4JgeEIwF0mppcBfe7Gv+um1Pvf67lgixSt9EAQO3b2zOyGYPQeTIAiCxSOsMorYGXOyqirFnxjWXdmbcefwFtc8/4Rh/Qy8xb/SfsMeKwgCS6gyiqzgQFWGtaqwnmmp3loqWPVHUwevnFWGZ6GRE3aprz1+STCW2q2NZvLMsguintWGF98DfLVO6VxC8AezQQjmIjaIUXIztr4NVvB5XrJWmHjCdK/T3mTG7Ak+T59tXVs943lLyQvZ+nqc6NR55SNNnQ3+tLpvCwu/yzVOnS3LoU5D7omW8GI42765UHsll27vmJ4DR+nJUXIKafP6UjpWzmZzf982hz2Xu3rFJ2tvERuP2YvNbJ1GnnDK1mmkNCHIldP9WPLbKa0J5Wj6VbUnBHORvfQupvsHdZwB7ip3WTqey5SDYCoIwRwEQdAhYvGvFbk3yUToOIMgGJ1uqzKKgRdF5AQR+byIbBORO0TkbXX9ESLyGRH5Vr19YV0vIvIHIrJdRP5eRF7W5oQWPV9tEATB84K5yd/S0yQi7j7gHap6MvAK4GIRORm4FPisqm4EPlv/D/BaYGP9dxFwTZMTmRvwFwRBMH4mWDCr6kOq+pW6vAu4E1gHnAPcUHe7Afi5unwOcKNWfBlYKyLHlY4Ts+QgCJaObs+YW+mYRWQDcCpwK3CMqj5UNz1MLxTwOhYaV+yo6x6iBdaqqhQpzsNLrgG+1RZOe66ulJPPe5DkZv2eqZFnihTu50EwbrqtY24smEXkBcDHgber6jMi8nybqqqIaJsDi8hFVKoOxGkv2QMHQRAMzxRYZYjISiqh/GFV/URd/YiIHKeqD9Wqikfr+geAE8zu6+u6BajqJmATwEEiGtHlgiBYWrorRZpYZQhwHXCnql5pmjYDF9TlC4BPmfo31tYZrwCeNioPlxSP2c6MV5i/lS3+Bi0ixuJiEAQVk69jfiXwy8DXReTv6rp3AVcAN4vIhVTpxc6t27YAZwPbqfzF3tzkRPrjTuT0pp7nrG1vGuRoHM9KT6i38bLt7vM6CKadCdcxq+pf46uBAV7l9Ffg4rYnkoTUswN7dUvfPIxgDWEcBF1gwgVzEATBdBKCOQiCoENMgVVGEATLizVzOtpp/45TLoXqHNU+vn+MQXRPhReqjCAIRuT+TDkYAR3P40JEzgKuolrbv1ZVr+hr/x4q7+i1dZ9LVXXLoDGbxMoIgiCYPuYb/g1AROaAq6liBJ0MnF/HErL8LnCzqp4KnAf8t9KpxYy5Y9hQfOnXPdbUeRlM1pg6m60kUcpZaPFSU0HvVdR7Jbb1tt3LIpNrn3PqvHvCvrKXstn028UvFekc9jh1UE6G4plh2gwkT9Vbm7jYWjOlnAa5lGTeb1lSZVimIiWZMq6TOh3Yrqp3A4jITVTxgrb1He3wurwGeLA0aAjmjvGVTDkIgjGSvNqacZSIbDX/b6o9l8GPDXRG3/6XAX8hIv8GOAx4demAIZiDIJg92s2Yd6rqaSMc7XzgQ6r6fhH5UeBPReQUVc2+aHRCMAu9aG8pklwup6eXILVN1pNSRLg2Ltrp9TgXqS5ho+PZV+r0Wewxvdff3PmXElVO7op5ECwBbcIu5mkSG+hC4CwAVf2SiKwCjqIXX+gAOiGY7VtFlzz7giCYUsanY74N2CgiJ1EJ5POAN/T1+TaVl/SHROQHqSJPPMYAOiGYgx6HmXJa1Ftr6uzi3xH19khTd3yh3Rsrt3ho+0rqZF8PVplyf7ATKK+4zTnlXMDrdFxvRdGSe73wXmu8wCVe8O7+cT1yK2neuXzH6ee9Ctnjm9v40fpY95nmu005KTwfNnWLsXho+5Q+CoW6ZVk8HMOgqrpPRC4BbqH6xa5X1TtE5HJgq6puBt4BfFBEfoPqkfCmOnRFlhDMHWO9KW+ot1bYWguNlJnAClBbPrTeWlnple39v9uUHzHlFU9X29wN5MkSK2ObBnzKjV8KTtX0+Lk3sjbPkHSsnN9YKdBWovR2aH+rb5pyEsJ2ad8K4SR4nzZ1Vgin87bC2HNGaWJVMrHJGpRxqTKobZK39NW925S3UQWDa0wnBPMKFs4UYcpNdfqwN/B2U75rqU8kCGYFxbe37AidEMzzlKPKTTNdf3AEwVQyphnzYtAJwWxnzOn1O2cdscJpt698SUX4nUx7ojS+7WP7HmLKydrCjj+qkLUPqN1Onf1c3oKp9/qZe2Uel1XHoDGCoJOMb/FvUeiEYLYz5lmeOXcZL4mtt05mH2zWTDA9cK3KynugenV2/EOdOtt3GHNH6D2E7IPXG9/W58wsm64zlnTQtv1EU35xvbV+v4cdZf7ZUG+PMHWlxQfvi7NP9NLswKuz+z2XaU8zCbu44Sm/zfgyrpluzJiDScebXXc3aOIMsjNTDnxixlzGOph42Ad7aRXYc1ApOZjk8M5psVekPYaxbrDYmav3vTRR63iUPpc3e87NMpveIzmri5LlnGeV4X2vufOw3+Eep877rnLqtkTuGk3YSaQXlyQ3Sy+9nXh13nfRxHLQs+yx7Cm0e/Ula5o/yYzVihDMZUpu6yVvPcs0OqiUTHdLTON3MmtYIZsErxVg9hpJpnFWe2CF7LxT55Xtg8fiqXvaqHKsOsp7YKxw2seeNLldrIwlpxOC2eLNXJb7wdZmIbLN/p02vg86xbOZcjACHb5xOiGYS6qMEov9/ZZsqjv84A2mBLuOl2aR9oG9HDFWBo3RecboYLIYdEIwz9FzGy692nixbj1v2KVMm2OZ2As16DRPlLsEbenwzdoJwbyPhS6lQRAEi0rMmIMgGBWb2WZjvbXBpzzLpdxbZVoUtOaOXhCjkumyrS8FLspZdTTta+ts8KahCZfsMl6sDMvEBUgJgjETmW0WgZgxD2bWY2UEQbDEhB1zEASLTcmOuOT+nnPcKR2rqUVSybKpTTjXu72OwxCCOQiCxaSk7gv3+T5i8S8IgqCDxIw5CIKgQ4RLdjP6w0aGpUUQ9Hi5KZ9SbzeYOms658Wv8Mzd7IK7vYe8OODPZfqmcXP5AUvtXs7AUmyYbzh1rYnFv2aU3ESDYJa5LVMORiB0zINZQS87c7JnzkW+SuzNtJciY63s28LC36cUvrEU06O0Im7xwjc+Y8qeob8XCjKXoDnVD5O1uH8/j3iIBhNLzJjLzNMLVbhrUMdgrAwT2zkIpoKOC+aSySIicr2IPCoi3zB1l4nIAyLyd/Xf2abtnSKyXUTuEpGfXqwTD0Znv/MXBDNBWvxr8rcMNJkxfwj4Q+DGvvoPqOrv2woRORk4D3gJcDzwlyLy/ao68J4Xeq/7KYh2Tr1QyjRRCqo/zkVFr93LFmLVG6XEsF58g5yqwouUVzq/UE8EQc0k65hV9YsisqHheOcAN6nqbuAeEdkOnA58aeAx6K3UdtEQvk32BG9FOoRdEHSMjqsyRtExXyIibwS2Au9Q1SeBdcCXTZ8ddd0BiMhFwEVQzZi7TId/v2BGeL0pv7reehHnAOaOqQurTaWNEuatUHsJFHNZsi1pNfoZpw7YU49hJyyl/IYlL8YTnbqh6PCNPaxgvgZ4D9Vz5z3A+4G3tBlAVTcBmwDmRHTI8wiCmeCjmbLLI33b4ECm0SVbVZ//yUXkg8Cn638fAE4wXdfXdUEQBN1i2mbMInKcqj5U//vz9JxxNgN/JiJXUi3+bQT+tjTeyroz9FJMWU+mtX19wV9k6y8nvMDbOdtebyHOvtG1SWPlLeh614I9Z+th9axT53lQ5Y5ZOr+IbR3MLJPuki0iHwHOBI4SkR3A7wFnishLqT7evcCvAqjqHSJyM7CNKmPUxSWLDKhu8KfrchI2dppt1WP9rtvgC+OcVYf3WwwruNK4OQeSRM5BxQtp6C0elpJu5q6vUZNuhuANppZJX/xT1fOd6usG9H8v8N42J2ED5XtrDJGIMgiCsTMmHbOInAVcRTXHulZVr3D6nAtcRvVI+JqqvmHQmJ3w/LN2zGF7GwTTx6pMOb112rdG7w3T7vO1cZzQmGbMIjIHXA28hsoK7TYR2ayq20yfjcA7gVeq6pMi8t2lcTshmK0ds6eq8NwTrbAO1+Ig6Daeim5ZGZ8q43Rgu6reDSAiN1H5c2wzfX4FuLo2KUZVHy0N2gnBbIkoc0EQLAnjUWWsA+43/+8Azujr8/0AIvI3VPPIy1T1zwcN2jnBHARBsOi0s8o4SkS2mv831X4YTTmIykLtTCoT4i+KyA+p6lODdgiCIJgt2qkydqrqaZm2Jr4bO4BbVXUvVaiK/0slqLOhtTsjmJvGowgVx2TirR20iYOdyrl42KV1CM+m2zuXUrxtu18TW3gPz0zSu65t+4+Z8j+rt6eaOuuS/d0pEtgxpnKtUz7U1FnHAO/LtIrheafezj6tTaeXDsUzvM85C6SxzDHFGvaPwniEyW3ARhE5iUognwf0W1z8D+B84E9E5Cgq1cbAZN+dEMxC74ZI14q9KD074dyNkMp7Mu1pv72F9v5ywtsvFzvVS/FjSZ8rZ2edyp7tsj2/SXAgKa0dlL7rWefTmbJLElz3LM65TAVjcslW1X0icglwC9Wtfn3tz3E5sFVVN9dtPyUi26gu9d9W1ccHjdsJwdz16HJBEEwhY5p5qOoWYEtf3btNWYHfrP8a0QnBbFNLNU3N1OZhV3LT9uI9W0ozt9KMOecyXUqZNU0z5iDoFJPukr1UlNyL+/vlVBmesCvFpygdq43gtedSEoLeZymo14qB8kMYB0EDJt0leymwLtmjkgTuvFNny1ZlUlqssTP2UpAiW1dakPL62f0jQ0kQLCLTFvazy5QiunVdh+0teq7OtJdoaqlQSl1l60uCPR4MwUQQM+agDfZaGRTYKQiCEQnBHARB0CGmMYPJYpBe0ZMdcy6ztPcqf6jTXlrcywXaX47FtVygfM8qw7PaKKkfSsdvcv4dnlwEQXuUhTdWx+iMYE43/q5lPYsgCGaGmDEHQRB0iFj8KzNHz30/WSBY1/7VTjmn6ljl1HnmclaVYccvxU/wzOFyD96mMRXsudroJ4/V26dNnc0Sn9QednHQpoH3chZ65n4l9UeuPiwwgokldMxl9gMp/l3a3p/pGwRBMBY6PHPohGAOgiBYUkKVEQRB0DEiVkYQBEEHiRnzYGw8Zi8wkF2oK7kGe/tbRtX3d/i3DKaYl5nyD9TbdabOLmCnxeScfX4KS2DvBWvS6y0q5+LYp7I1c/W8V4ddKE7namXA/y6M1YhY/Ctj3ypKgYFKdPjtJFhEhsmAU8ogMuyxmo7VJu7J0ab8onr7YlPnJSspOWZ54XShJwxzjk2WUqAtL8FJKSlCKcLiWARz7kQ6QicEs43HnLaeCZwtl8zlLN4NmAsVWvIYHOa3tMfyLlDbvssp25mLDcLkPcy82M3DBiaapHChwxx/2HMe12dtM84tmXIwJDFjLiP0hFP6rjwBBD1zupztrRf2cxhhZcueKiXXbkl2zLmg/Olz5eykvetmVPfw5RagQdAJwiW7jLVj7iIl9UioT4JgAokZcxAEQYcIO+YgCIKOEYK5Gf1xI0q6VMswbyS5BT9P35xbPU/64tKCYW7/pAKJ1FJBsAyEKqNMEhhLla1jktJNBUEwZmLG3Iz+WeWKTFvJ9rM0iy0lQ/WOlTPU39vXD3wLkFyW7ZLVhRfdzpY900LP0eCwTPsaZ/9cVL5S1L5Uby1UDjFlL5KeJdWXMpLnfn/P9M+zpmljO5yzefZsay2lqIPDvOnY78UzGbWUbJbTWPbtzDMpzdkWe/dAzsqpZPjQ1PLI9vuFwpiNCJfsZvRfrKVs1EEvLGgQBEMwyTNmEbkeeB3wqKqeUtcdAXwU2ADcC5yrqk+KiABXAWdTeXa+SVW/sjinHgSzwz815ZPr7fGmzr4J9Yc36C/vceqsjNrt1OXUfd6M1u5nU6UlvDdUO34pZvg1mXNpxRQ4mHwI+EPgRlN3KfBZVb1CRC6t//8d4LXAxvrvDKrv8IzSAVbQe91e2bftx1Mv2Jx/6fV1hVNn61dm2kuvfN5C4YpMu+eZZy/A9Fk89Yfdr6QDt8e3fb3rzlPL5G5Qz2OxtFBp8Vx6S16GObVR6Zil/Ibpdy3tU1oItuWSS3dJ1VFSy1i+acrbnH6L4dHZ4QnleOjwBywKZlX9oohs6Ks+BzizLt8AfIFKMJ8D3KiqCnxZRNaKyHGq+tCgY8yzdIt+wWzSVB3WZiF4WBWbp8Mt8cSQxwoyTOni3zFG2D5ML4bKOhYmH9lR1x0gmEXkIuAiqFyygyAIlpQJV2UMRFVVRHSI/TYBmwAOEtH+XHzeKrQlt0rs2f6O0w66ww/ZYIrxLGusCs+LJNfGaqOkQrLkrDkG7d9EReTh6bBvLezTiCm1yngkqShE5Djg0br+AeAE0289C/OLupTCfnoLCCEgg1ni2Uw5GJKOqzJyZqMlNgMX1OULgE+Z+jdKxSuAp0v65X7mnL8Vzp/Xb45q5rCK6gk96C+3v/0LgmCK2d/wr4CInCUid4nI9toYItfvF0REReS00phNzOU+QrXQd5SI7AB+D7gCuFlELgTuA86tu2+hMpXbTjXRfXNp/H5Kq+vD7B8EQbCAMZnLicgccDXwGqo1tdtEZLOqbuvrtxp4Gw01MU2sMs7PNL3K6avAxU0OHPgc7ZTXmrojTHm1027LSe9Y0kXadqsLXOvU5xIMeNjr3jM99EzzSinD7P6l171hVIht3pRypoNzTrslfS+5z+Ktk9xryg/WW+tgZK02UghdqwL0ki20yRoylaZ54zno6cB2Vb0bQERuorJO29bX7z3A+4DfbjJoZzz/gorHMuUgCMZIu8W/o0Rkq/l/U228AL4l2gLfDRF5GXCCqv5PEZkcwWwdTNIscE2mrxeTwYslMWzMBG912v5+dnXccxTwZpGlWWIpFoed+XiZXbzkl7a9zcwod61O9MwoCBxaXIs7VbWoF/YQkRXAlcCb2uzXCcFsHUzS9uFlOpcgCBaXYdRFdh/PSqstYzTKKFmirQZOAb5QRazgWGCziPysqtpZ+AI6IZiDIJgdhlnMXwyT4zH5l9wGbBSRk6gE8nnAG1Kjqj4NHJX+F5EvAL81SChDRwSzzZJ9mKmjrw58VYGt89QD3j5NVBleH09tUVrkyakqvFgbT5tyenvIqTK8RZxZzZIdBG0Y14xZVfeJyCVUycvngOtV9Q4RuRzYqqqbhxm3E4LZ4uk9rWAqRbMaVNe/n0cImCCYDcblka1j8Dw7AAAU40lEQVSqW6hMhW3duzN9z2wyZicE8zywa7lPIhgJz23ei9SXi+S30qnz3kRKEd3a9C29EVlyEQQT3qt2afLQ5vx+0ZRPrbcbTJ0N+5nGshMaa06XrH0eN3W7nHavrr9+l1NXeqvzQny2CbEwDh3zPOUg/stJJwTzSnra82PrrbXKsDdrUmvkbG+9WBulGzAXlL8kGNJx22Sf8FQluVCb3kXrheLMxQXxLmrPQiMn4DxhUzpW7gZrqissfRe579qzxinh2Vk36evFCLakc7C/dclOORdLInGDKV/bdx4wurAb9nsr0dS1eDniCXU4hlE3BPN+ek/0J0xdrm8/oZ4Igm7S1Xuv46EyuiGYLWmmbF/NrOAt5YY7OFOfaLPKm151cnbApRlnSR/ueYh9rykf37cF3wvwSFNn3yS8t4s5p29utlZaCPXeSnL5D723g1wCAq990Hn0j+vhfdel/I+578I7F2/2Wwq0n7sGvaQE20353nr7oKmzqoan+rbgLyB7Kgd7/NxvNcz6ThcnTyGYC3h2zLNKBEQPPOxEJT1wc5lvShOCYQSnpcsCrSkdzyzVDcEsHDjLaJNZ2mMaLh4PLzO1NSe0uvnDnTrP9NC76fuPtdqpK72deOeaW3Dxsmi30YuWYm2Urqv+cXLHt/W2r/f2UNo/p+P2ztXOWNNCm3XCesJptwtyEStjIR0Px9wNwex9SdMqWEfFu5nsDRgek0HQjC7LmE4I5iAIgqUkFv+CIAg6SOiYgyAIOkTMmMfAMGme2phFlY67XO7dafHMLsh55nKrC+3HZtrXOHVHZ/qu7tuC77knhzg7Qe9D5IKYrOrb9reXVhoTuR/+EKeuZAPXJvHaMBdZKR5srq8Xcs1S8sZJeDFi+8uJnG1d/zHBj9C/u1elZvw0lD3VkrPMeufwbQnBPAaG+QJL+zQZc7lXbdPxrT2qLd+7dKfSnN2Z8s6lPpEgyBNWGUEQjMxGU04OR7m3l4SdedoZaZrE5ly6Sw4odqympnXDzk4908RxxMqA0DEXEfw3WQbUeV5llpLdZY7F8Goq7R8EJb6VKc8K457dhiqjAYr/lA6CIFgsQjAHQTASVm1RivsRLtllwiW7IYMsL6bhQgiCUYh45eOny3KlM4K5y19SEATTRVhlBEEQDMkwPgxNiMW/BlirDM88po1OzLPpH1aXtBw/3NFO2auz5SNM3VqnbOMxH27KXsS4XGqnUmaYlU5dKfGs97vkYgB7WUHaXCOl66Lks2HxrlHvXEe1DLLnfL8p31dvc/GYU8ooLwUUjDcec2JUHXabxBjjInTMBaxVRmLY14wuPwWb8FimHATB+IgZcwNW0PPYTbO4XD60NEvJCW7PhdPuX8pj1yarRak9jevZYdv2yPgdBEtPl++PTghmpecwkl65SqmjvNdou1+ufdCYub65gOXDxNLw7LTtMb3wBd5rpj3uYqf9ye0XTA+l+yUXQqTU1zPtK4U98dRltu6rhf2bEIt/DbBfkvdl5dKoB83I3VTpBrFxh7yM5NDTU+cynKRy7iHn6aBLOmzLMBnJvXhFuePP9/WDvL7d61uKQWTbUwiR3ITBewhuMOXkkm3XFqydc/ouc/r49HuX9Ma51FXeRMFOKKzLdEnf7um7PRlgz+Uip70tocoIhsYTJl7ANitYD3b6WgHjJWvNxVzwAr0dmmn36jwh6M2m7Pi5mZUnsL3Fv5wA8YSgV2eP2ebG9c4ltziZyrljeZ/FCqYUyMpmq/EW+koB4awA9VJLkWkvJSS2QtqLldHmrW6xrDL6z6NrdEIwWx3zyr4tlGcgbSI1llaULU310SUdtCVn1eDh3aDeDRg66CBoR8yYGzBP74k+61mygyBYfMIluwFC85jki/2U82axuVluKltVQsl2ts0rb5qxh9VGEIyfLl//IwlmEbmX6o16P7BPVU8TkSOAj1KtV9wLnKuqTw4aZw44si4npwmrq7Q6UKvjTNgFqbRfzkSuZLUxzI9VWt0tCV57/OVIQ19yDugfYxBdvtiDheQSxHi699zaw+r+jvj3k3eP2r451aXXfp1zzLbMglXGT6iqzU9xKfBZVb1CRC6t//+dQQPsp+et9JSpSwzrBTiINrpei9fXM9cr7WPrc4IxZszBYlLKJjXNzKKO+RzgzLp8A/AFCoL5IHp56ZIb8RrTbp+2nvrAM/EquQ7nVA5evZ1NeHbGudRpnoNJyUHFc5215GYuHp6qxJsl5dRIbR54aXZeslFt83biWVjkZlZ7+vr1c7DTXno4l1Lq5VyWS+qwpgvEud8tXSM2zZi1sBiXy3UbN+lJnAR08ZwSowpmBf5CRBT4Y1XdBByjqg/V7Q8Dx3g7ishF1CaJQi8WwP1e5yAIgjEyzsU/ETkLuIrqWXqtql7R1/6bwFuBfVSuGG9R1fsOGMgwqmD+MVV9QES+G/iMiHzTNqqq1kL7AGohvglgLtMnCIJgsRjHjFlE5oCrgdcAO4DbRGSzqm4z3b4KnKaqz4nIvwb+E/D6QeO2SdB+AKr6QL19FPgkcDrwiIgcV5/0ccCjoxwjCIJg3KQZc5O/AqcD21X1blXdA9xEpc7tHUv186qatE1fBtaXBh16xiwihwErVHVXXf4p4HJgM3ABcEW9/VRprBX0VneXKm1OG51fDk8H7bXndJXeU3EWdX1BsNTY+DwNOEpEtpr/N9Vv/ADrWKiB3QGcMWCsC4H/VTrgKKqMY4BPikga589U9c9F5DbgZhG5kCp07LltBvXMY0qCtU3EuJJg9yxAbHsbN2FvfEs679JDqDRWCOMgaE+LCdlOVT1t1OOJyC8BpwE/Xuo7tGBW1buBH3HqHwde1WaseXorzWm+nwsg4+FZJ8xl2kumcd4DITcjT2OVgrcPY90AfmCnkh2yJdW3eTsIIR7MAmM0l3sAOMH8v76uW4CIvBr4d8CPq+ru/vZ+OuH5Z0lfVhvX7FlKVOnFBckZ/HvONl5fm9Uk1/dQp90z52oT0rH0kLAPqdKbSBtnmYSnNsp5cVpWOO3euXgP2f6yV+eZHnqhNu13Ys3h5p26UkQ47/yanH/pDc576/T6lqICWp5z6toyRsF8G7BRRE6iEsjnAW+wHUTkVOCPgbPq9bginRPMwWA8wWNvlFl6SAXTw6SmllLVfSJyCXAL1bPyelW9Q0QuB7aq6mbgPwMvAP57rfr9tqr+7KBxQzAHQTBzjNMlW1W3AFv66t5tyq9uO2YI5iAIZo5ZdMlujc2S7enPPEqWFCWX61EtIXJjee2j0uULKFga7DWe9P2lBAelIEW59YT+jPX9xy/FSvcoLeCXxrHt1zQ8Zoku31edEMwlPKuLNql8vAupiW1xKadfGrdNLIwSdpEmLd2WFmGGzQ7h7ZOjyxfxLGC/fy/qYNCOiMfcAKEnBNOXZYWdl3ssJ+y8GfMwmbFtn9KMuU3uNm8mn5vde7nb1pqyN3Na7fT10lHl9rftXk4527cU+KeUqNPLCZj7LksPyUQptVSbnH+lh7dtL11jFs9+3VLKv+dZvtj2pg/0nFWGV5e7X7zgUZ61S5vJhWWurx9UXmzjoMuTjU4I5nkONI+bpEwmw/7A3n424tO6emsF5LGmnASrFda272F9/Wwd9ISsJyD7x7X7JdqYvqVxS7P70gMzFz0ukVvQ8b5rT/DmTPRKpnmesMoJ9qZC3O5/tyknNzOb88/G8U5R53JxvJO5WS7PX1P7+f76QXVds58PHXPQigcz5SAIxscsBMofGZuMNc3CSq+UuS/Ve8309NFNnpaePnqYDL6l1FQWOyP07JSH8fyzdG3mEgTLReiYWzCMh1cseAXTjlVRJRWXVXsd6fS1+3hqKavi8tYOcvr4UtgCnHaLF6A/t3+aqNhxzsv0bUOoMhrgRXqyF00pFsYhTn1uEclbOCm54ZZ0iaUHg5dpA/zUUV77rGeaCBYK3nRv2OvKZjNJ+uRc0gkvw4nFu0Zy+mgPb1E0t47h7eMxUnziDF2+1jshmIWewExCshRbICcsvUWmNjkDx5lTr7R/EDTFXkNpomInLyULkTaCrbT4Z4+VHgK5e8ib8XqRIXOCuUVozlaEuVwDrFXGYlhjtEm2Okyc5jYPAY82zjTBbHJ3phwMT5fvq04I5sWmzQ8w6o81zP5dvkCCYBqZJ6wyiggH6p08F1Iou1x7+5QyX+esJko63v5z6j9WU7VIG1WLx6iCfZhZfrC0lBJHWOI3bEaXv6dOCGbPprDLT7Nxs9yftcsXaFBhr5GkY87FyvBCBXgel6UFudzkZ9Q1G4/cPp6X5JcajjmI0DGPgdDBDodnoeJZu1ivPmtWdYjT13P5zh2rjbBI9TmzrIR90/HioZQWmXK0WYdI5Nyk03nlLIMSJft0e/4nm/IP1dvDrb2cdQldU29zkjudbG5BxTM3sos/XgT+vZn2VG+j23uSN+OzvX/3gbvY63IUuiw3JkIwd/kLbMJyqQo8N+eSS3PQTexDMMnTvY/06uZNeVLdpJeSsGNugPX8y7UnSra73szJ0jRmgjfmsAybVidusCDxVLlLpxk1jO+o92A/4ZLdAOtgUoolm8jZN3pCquQ0UhKcpVnmqPGYQ3AG086ollGLIUS7fN91QjBbkhC0KqmuZ3nu8g8cBMGBxOJfA0qvFZMq+EpxP4IgWD66fC92QjBPK13+4YNglokZcxAEQQfp8sQpBHMQBDNHWGUEQRB0jLBjbsAcPQN6zwPMs2ss5XbL2QNHDOMgCEIwN2A/k29AHwTBZBGLf0FjjjdlL0u2F6vChkFYY8qH11sv5gX4IRNWZcpzTt+SOWBuLG9/z0nHC7xjx89FCPRIN2Eul+QwHqFtMnmUZmfeZ7V120w5xWN+wNQZj2wer7dPmzovS7YX8gL88BVt3kCHTeU2aP9xEzPmYGjSzWSFiVXhWCecQeTClqb6nGejHT/dpLnMMiUVU6KU2DYnONPxPS/Q/rGa4j0YcgKo5BJcGmtUhhGcuc8yCYJzKYgZc4EV9GaCaWtvQC+nH5m6pjOf3MXVpm8phU8pcaw347JjPlhvbe62iKWx/LTJtD4uXmTKG506LzGrDXxkowqmNzAbn8Z78Ni3nFJMckubNwXvHvGyGNlr+J8XxmyCl2e0S3RCMM8Du+ryrkEdg6ADLMeDzEst9bklPH5OrbPCafeC+ufUaYmcWsqLnTMOZtbBRETOAq6i+s2uVdUrFutYQTDtLHeWmZIVUy4cc2IxcnmOSpffFBdFMIvIHHA18BpgB3CbiGxW1W2D9wyONuUT6u2Jpm6DKadX2Y1OHcC6NM2wA5zgdLaD2ndi2zetStqA7HZV8qAjnA52sPX11r5gr3X6rsq0p3IurL93KXtzM4vdZ1XfdtBY+/q2uePnbq+03z86df3lxNdMeUe9fdjU2aW+JBqtrdM/OmW7ZGj66rerrZWmdvhnnHrb9wlTTvX2VdiW02F3+6fy/P5G8stmRmaci3+lSaiIHALcCPwTqrXZ16vqvYPGXKwZ8+nAdlW9uz6xm4BzWLi4/Dw255+nvxs1J16JYaPXlRasvPFL7Xa2cVe9zT3Nit9Fupi3m31Mmc9nBh6KJ/q2kD/zIFh+xqHKaDgJvRB4UlW/T0TOA94HvH7QuIslmNexcM1qB3CG7SAiFwEXQSWYkwzxFhYW23Vy2CenlyEkCILuM0aX7CaT0HOAy+ryx4A/FBFRVc0NumyLf6q6CdgEICKPPQf3AUcBO5frnBaJafxMMJ2faxo/E0zf5zqx3GUw83DLs9X30oRVIrLV/L+pll/QYBJq+6jqPhF5msp4JvubLJZgfoCFGsr1LLSHX4CqHg0gIltV9bRFOqdlYRo/E0zn55rGzwTT+7lGQVXPWu5zGETJYWpYbgM2ishJInIwcB4wBpV9EARBp2gyCX2+j4gcROWg+zgDWBTBrKr7gEuAW4A7gZtV9Y7FOFYQBMEy0mQSuhm4oC7/IvC5QfplWEQds6puAba03G1TucvEMY2fCabzc03jZ4Lp/VzLTq0zTpPQOeB6Vb1DRC4HtqrqZuA64E9FZDuV2dJ5pXGlILiDIAiCJWaxdMxBEATBkIRgDoIg6BidEMwicpaI3CUi20Xk0uU+n2ERkRNE5PMisk1E7hCRt9X1R4jIZ0TkW/X2hct9rm0RkTkR+aqIfLr+/yQRubX+zT5aL3xMFCKyVkQ+JiLfFJE7ReRHJ/23EpHfqK+9b4jIR0Rk1TT8VrPGsgtm49L4WuBk4HwROXl5z2po9gHvUNWTgVcAF9ef5VLgs6q6Efhs/f+k8TYqC5vE+4APqOr3AU9SuZ1OGlcBf66qLwZ+hOrzTexvJSLrgF8HTlPVU6gWo5IL8KT/VjPFsgtmjEujqu4BkkvjxKGqD6nqV+ryLqobfR3V57mh7nYD8HPLc4bDISLrgZ8Brq3/F+AnqdxLYTI/0xqq0L7XAajqHlV9ign/ragsrb6rtpc9FHiICf+tZpEuCGbPpXFdpu/EICIbgFOBW4FjVPWhuulhFoZdmwT+C/Bv6cV9ORJ4qrZXh8n8zU4CHgP+pFbRXCsihzHBv5WqPgD8PvBtKoH8NHA7k/9bzRxdEMxTh4i8APg48HZVtUESqQ3LJ8ZGUUReBzyqqrcv97mMmYOAlwHXqOqpVAEmF6gtJvC3eiHVjP8kqkCthwGddj0OfLogmFvF1eg6IrKSSih/WFU/UVc/IiLH1e3HAY8u1/kNwSuBnxWRe6nUTD9JpZtdW78uw2T+ZjuAHap6a/3/x6gE9ST/Vq8G7lHVx1R1L/AJqt9v0n+rmaMLgnlq4mrUutfrgDtV9UrTZF0yLwA+tdTnNiyq+k5VXa+qG6h+m8+p6r+iiub8i3W3ifpMAKr6MHC/iPxAXfUqqlCNE/tbUakwXiEih9bXYvpME/1bzSKd8PwTkbOp9JjJpfG9y3xKQyEiPwb8FfB1evrYd1HpmW8GvocqvOm5qvqEO0iHEZEzgd9S1deJyIuoZtBHAF8FfklVdw/av2uIyEupFjQPpkql92aqycrE/lYi8u+pgrDvo/pd3kqlU57o32rW6IRgDoIgCHp0QZURBEEQGEIwB0EQdIwQzEEQBB0jBHMQBEHHCMEcBEHQMUIwB0EQdIwQzEEQBB3j/wPhAN2UCdACXwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAEICAYAAABhxi57AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3X30ZVV93/H3h2HIhAcFHAM4MwZUksgyFewsJEsbqagZjQGTpgTMA2aRTtIFrUbThNjWp66upW0isWtRm1GINCUi8aFOdaoSgqF2VcKgiDyomRKUGXlwFHyADsMw3/5xz2H2/Nzn7nPuPfd37u/yea31W/fcfc7ZZ9+H3/7t3znf892KCMzMbPkdMnQDzMyerNwBm5kNxB2wmdlA3AGbmQ3EHbCZ2UDcAZuZDcQdsJnZQNwB27KTFJKeM6/1mS0Xd8BmZgNxB2wTk/RcSZ+V9JCk2yWdXZV/VtJvJdu9TtLnquUbquIvSfqBpF+RdKaknZLeLGm3pLsl/Wqyf6f6Zv26zfpy6NANsJVJ0mrgfwBXAK8AXgx8XNLGcftFxM9KCuD5EbGjqutM4HhgLbAOOAPYJml7RHy1a31mK4VHwDapM4AjgXdGxN6I+GvgE8D5U9T5byPi0Yj4G+CTwLk9tNNsbrkDtkk9A7gnIvYnZV9nNIKdxIMR8fCSup4xaePMVgJ3wDapbwIbJKXfoWcCu4CHgcOT8uNb1HeMpCOW1PXNanmS+szmnjtgm9SNwCPA70taXZ3H/QXgauAW4JckHV6Fh124ZN/7gWdl6ny7pMMk/SPg1cBfVuWT1mc219wB20QiYi+jDveVwG7gPwO/ERFfAS4F9jLqGK8Erlqy+9uAK6voifo8733Ag4xGvVcBv1PVxYT1mc09OSG7Da0aPf+3iFg/dFvMlpNHwGZmA3EHbGY2EJ+CMDMbiEfAZmYDWdZbkVdJUR9QmfW5sqHk/i9oal+p3VryuNxKx01fa5dtnyyaXvMs3qt0RNT2d2QlfCZt251u9z3YHRFPn+a4mzZtit27dxe3u/nmmz8dEZumOdYklrUDPhQ4oVpeXT2uStavYn48nilL25f+oqxeumHDfl1eX5//mpSOm77WLtv2ZX95k0E1veZZvFdrGpZrue9Fn+9f03fh8UxZF23bnW73mdHdkFPZvXs327dvL24nae20x5qEk/GY2QILYN/QjWjkDtjMFlgAe4ZuRCN3wGa2wDwCNjMbiDtgM7OBuAPOyl1NTa+AzuLKeFNkQe5YpfZNG7FRukKeW9+0T9v3qs3+peOWjtWl3W3r7FLXLDS1rxSp0vZ1pfUcnSwfVT2mUTaHtaxzUun7+1hmfVrWJUpikiiIfrgDNjMbkDtgM7MB7AceHboRjdwBm9kC8ykIM7MBuQM2MxuAR8AHWXqVtHTVdH+mrGm/0hXypmO1jWhoOv6qzPrScUr5I5r2K7Ult76Luq6hIhf6in7p8l0o5T+YVRty0vwPbb/jTe/ZJJE2XSI+ury+IaJXRua7Ay7+nkpaI+lvJX1J0u2S3l6VnyTpRkk7JH1I0qwjZMzMOtrP6Fbk0s8w2gyUHgVeGhHPB04FNkk6A3gXcGlEPIfRZIpLZ6o1M5sD+1r8lEm6QtIDkm5rWC9J/6kalN4q6QWlOosdcIz8oHq6uvoJ4KXAh6vyK4HXtHoVZmbLpj4FMX0HDHwAGJcz+JXAydXPZuC9pQpbnSqUtErSLcADwLXA/wUeioi65TuBdQ37bpa0XdL2ec/7amaLpr8OOCJuAL4zZpNzgP9aDVo/Dxwt6YQx27frgCPi8Yg4FVgPnA78VKsWj/bdEhEbI2Kj5z8ys+XVugNeWw8Uq5/NExxsHXBP8rxxYFrrFAUREQ9Juh74GUa9+6HVKHg9sKtjY83MZqx1FMTuiNg448b8kGIHLOnpwGNV5/ujwMsZXYC7Hvhl4GrgAuDjXQ6cC90qhYNNu76UjKcpGVBdPmkCnllMSVSqa5KkOXAg2UpTgp5S6NLjme1ybemyfijTtmGSZDOlMLRU3b7lOLWXa/ckv2/T/g53t6wJ2XcBG5LnxYFpm7MCJwDXS7oVuAm4NiI+AfwB8EZJO4CnAZdP1GQzs5np9SJcyVbgN6poiDOA70bEveN2KI6AI+JW4LRM+V2Mzgebmc2poK//pyR9EDiT0fnincBbqe6pioj/AmwDXgXsAB4BfrNUp29FNrMF1t+dcBFxfmF9ABd1qdMdsJktuPm9FdkdsJktsPpW5Pk0eAecXsFte9W3adtZJQdpO2VRl/U5pWiDVDo1TCkyIZfYqEtbpjVJRMe0yYRmJfe+9JnAJxcF0TRNUO61lqaXosP6VO5YuSmL0u1yEU5N7avX5+qcznwn4xm8AzYzmx13wGZmA3IHbGY2AI+AzcwG4g7YzGwgjoIwMxuQR8CNDmlYLm3bdu6yLprqmjbkrW3Ski6JiUomDdmb9n1drjndJjXrMLXUJCFrubkCm0KzHsus35NZ35T4KJc4qYvce9kUhta2rv5T1voUhJnZQNwBm5kNxB2wmdmA5iG7dJ47YDNbYI6CMDMbiE9BmJkNxB3wE8SBMJv6MQ07OSyzz6zCzHLhOE0hSrk543Lrm0K/Sutzr7FLVqhSNrTcdqX3tZRVaznmIRt3/CbThpmVwqBmkVEubfNTk+XDMuvTMLXHx5Q1Hb9L+3MZ9SZVyng3ybyJ7bgDNjMbkDtgM7MB+CKcmdlAfArCzGwg7oDNzAbkDvgJXaMaSkllSpELTdtOkjSk6ViHFNYPHUVQa3r9k2h6z9rOlTfJPHF9mrT9s3ZEpqxpTrg9Y8oA9mbW5+YSbIp4mSQCqen9a5uQqn/zPQIuvm5JGyRdL+kOSbdLen1V/jZJuyTdUv28avbNNTProu6ASz/DaDMC3ge8KSK+IOko4GZJ11brLo2IP5pd88zMprHCoyAi4l7g3mr5+5LuBNbNumFmZv2Y32Q8nU69SDoROA24sSq6WNKtkq6QdEzDPpslbZe0fX7fBjNbTPN9CqJ1ByzpSOAjwBsi4nvAe4FnA6cyGiH/cW6/iNgSERsjYuPQFzjM7MlmvjvgVlEQklYz6nyvioiPAkTE/cn69wGfmEkLzcwmNt9REMUOWJKAy4E7I+LdSfkJ1flhgF8EbmtzwKVhSE2JRnJyw/U0hKaU4KZLvcspd/xcyFjTnHGlMLgu/3mU3uNx27UxyX9BXT7X3Jx2Q//n1bbNkH9fm/bPzSW4JrM+l+Sqqd6mhFWTfN5d3vfctjsmOGbeCu6AgRcBvw58WdItVdmbgfMlncroT8zdwG/PpIVmZhPrLwpC0ibgPYz+Xrw/It65ZP0zgSuBo6ttLomIbePqbBMF8TlGmSSXGluxmdnw+jkFIWkVcBnwcmAncJOkrRFxR7LZvwGuiYj3SjqFUR954rh6h/7v28xstuLx8k/Z6cCOiLgrIvYCVwPnLD0S8JRq+anAN0uVOheEmS22fu77XwfckzzfCbxwyTZvAz4j6V8wuqv8ZaVKPQI2s8UVjK4sln5gbX2/QvWzeYKjnQ98ICLWA68C/lzS2D528GQ8pSv3pWl80lc3i+li0nq7HCu3Pk2EUop8KE2Z1CUyoc8ogElupim1ddoBynJGPMz6ZqJc+5uS8TyeKcvpkviozfK09ebMbCQYtJ3fa3dEbByzfhewIXm+vipLXQhsAoiI/yNpDbAWeKCpUo+AzWxxtR8Bl9wEnCzpJEmHAecBW5ds8w3gLABJz2UUFfitcZX6HLCZLbYezgFHxD5JFwOfZjSgvyIibpf0DmB7RGwF3gS8T9LvMur6XxcRMa5ed8BmtrjqEXAfVY1ierctKXtLsnwHo/smWnMHbGaLbY6zgLkDNrPFFQwz/UxL7oDNbHEFB+ZmmkODdcD1H6X0v4M0JCM3X9W4eroqhfuk6nZ1aUvpv55c6FTT/vX6pmiatiFCXZL1NIXHTPJ+d3ktpf1L7ch9byYN9ZnFwKkUMpcm01md2S43Z1vpvZhUl3kT6zakCbVKibZy62cSluURsJnZAHq8CDcL7oDNbLF5BGxmNgCPgM3MBuIO2MxsIO1zQQzCHbCZLTaPgJul700uVKVLOFeX/Urn5fv8zGYVStdWl6xVJW3fl2kz002SrQ6WN7tULktebn0XxybLx1ePT03K0vndcmFquSxzTXMJ5kyaWa4eZKbHSn+fc/PS5dqV7nNVh+M38o0YZmYD8gjYzGwAHgGbmQ3EtyKbmQ3II2AzswE4Dni8/Q3Ls6g/1dfcZk1ySVP6qHecpqvduTnlmq7cl66s5+rPbdt0Bb20PlfnLLSZP6/0ubVtY5fv2suT5XqCsp9Myo5JG35U9ZjL4JMqhUmkcbJN/67X26QvJv3i7ikcKyf3xU/af9Ejhf3bmuMOuBi1I2mDpOsl3SHpdkmvr8qPlXStpL+rHo+ZfXPNzDqoL8KVfgbSJmxyH/CmiDgFOAO4SNIpwCXAdRFxMnBd9dzMbL70MynnTBQ74Ii4NyK+UC1/H7gTWAecA1xZbXYl8JpZNdLMbCL1rciln4F0Ogcs6UTgNOBG4LiIuLdadR9wXMM+m4HNnQ9mZjatRbkIJ+lI4CPAGyLie5KeWBcRISk7/XJEbAG2AKxp2MbMbGbmOAyt1a3zklYz6nyvioiPVsX3SzqhWn8C8MBsmmhmNqF6BDyn54CLI2CNhrqXA3dGxLuTVVuBC4B3Vo8fb3PAOkql7vlzyUVSpb8Quaia3PGg3zmySnWVEqB0SXpSmrMtl8gkN9/W6kwZ5D+Dpm1XLXlcur6OiErbV6q/SxjbJOFp086P16XektJg7J+nbTm7WnhJUvisZDmXreeItILqsSlDzqPV456k7OFkORdmlm6bnjt9iPHqL3y6z2OZ9anNhTrbWIBTEC8Cfh34sqRbqrI3M+p4r5F0IfB14NzZNNHMbEIrPR9wRHwOUMPqs/ptjplZz+b4HLADE8xscS3AKQgzs5XLHbCZ2QCcD/gA8cNX0UsJZJrk3tNpp7aZdBqcUl1tk7o0RRbkEuTk1jflZKmjEH4kKVtTWE4vpueOlYt8SJeb1ufamkZJ5KIgukypk5PbpxSFMW39kE+CVPwunJ88uah6fNEzk8KfS5afXz2uT8qeliwfuuRx6XIduvDdpOy+ZPkHyXK9ze7M/ul+aRjFvmS5Dp9I63wosz7ZZ3NPPadHwGZmA5jzKIjlnMPQzGx59XgjhqRNkr4qaYekbPIxSecmmSP/olSnR8Bmtth6OJMhaRVwGaOUzTuBmyRtjYg7km1OBv4QeFFEPCjpx0r1egRsZourvxHw6cCOiLgrIvYCVzPKCJn6Z8BlEfEgQEQU0zO4AzazxdW+A14raXvys/RG6HXAPcnznVVZ6ieAn5D0vyV9XtKmUvN8CsLMFlu7UxC7I2JjebOxDgVOBs5kFJpyg6SfjojGTBlz1QEf0rBcy4XzpP895BLzdAlbavpPpG04UdP63OefuzDbJfFQbk62tCwN7arDz9LQrzTMLC0/vLC+bRhaU0hcLgwtFwaWtp/M+i5KYWilbUuaEkLlvjfplGvZYx2bLD+jXviHSeELk+W6PA1DOzpZLv161/1CGlqWZvZJQ8rqbdJvRu5YaZjZnsxy+smn7av3S0PXvsPU+ouC2AVsSJ6vr8pSO4EbI+Ix4O8lfY1Rh3xTU6U+BWFmi6u/c8A3ASdLOknSYcB5jDJCpv47o9EvktYyOiVx17hK52oEbGbWux5uxIiIfZIuBj7N6J+XKyLidknvALZHxNZq3Ssk3VEd9V9FxLfH1esO2MwWV4+3IkfENmDbkrK3JMsBvLH6acUdsJktNt+KbGY2gDm/FdkdsJktLucDPiA4EIaTy/CVC+HpklWslCGty9xrpf1KIWldsqXl/kCnYVK59yInt0+qKetXKdtZl/W5sJrce9k0HVi9vstpuy6fcW3SDGvTZswr7f/sLyVP/qp6fNnHDpSd9Mlkg+dUj2uTsjRMLCcN86oznOWymsFBIWX/r3o1aWTaI8ny96vHpjnj9mTKHs5sO4vO0ukozcwG4BGwmdmA3AGbmQ3AM2KYmQ0kvfA0h9wBm9li8wi4WZcoh5Jpk/GkclEO6dX+/Zltu8xvVyorRVakF5Prdu3JlEF5nrZSMp1cxEQuGVC6bdP7Xpc3RWzk9mtKdjNu/7bz8E1zrJJJ9l/zuQPLp36lWrgm2eC4ZDh3VJULvCnzUd2ApsxQ9RcmHSGmkQ25KIZcfp10uSm8pT5ueqzHMuv7NucX4YrJeCRdIekBSbclZW+TtEvSLdXPq2bbTDOzCdTngEs/A2mTDe0DQC6x8KURcWr1sy2z3sxseD3NCTcLxVMQEXGDpBNn3xQzs56t9FMQY1ws6dbqFMUxTRtJ2lxP8zHH74OZLaI6F0TpZyCTdsDvBZ4NnArcC/xx04YRsSUiNkbExkkviJmZTWwln4LIiYj762VJ7wM+0VuLzMz6sog3Ykg6ISLurZ7+InDbuO1raWa4Omql9MenKUSpNJouzSnXpnxpGyZN2pKbfywXodP0Xjy25HGpXOhXurx6yXZw8JxrufKmBDulxES5fXLS43cZgEyS+Ci3/6T/jXX5vuaU2pp+xjdV07Bt+KsDZWmqnVxIYPq+5/Lb5EIZmyLDcstNoZK59V0SVs10EDrH5z6LHbCkDzKa52itpJ3AW4EzJZ3KqE+9G/jtGbbRzGwyK30EHBHnZ4ovn0FbzMz65VuRzcwGtJJHwGZmK9acxwG7AzazxeUO2MxsQD4F0WzamzOawl4mnVOsL13CpXJhYqlStrXS+ly40uqGbes60jAxMuubtA0PLIUoNZk2Q1ltVmFo037vvp0s19eO7krKcu0uhS82haE9ninbW9i2VFeTLnMk9sojYDOzgXhaejOzAXkEbGY2gJV+I4aZ2YrmEbCZ2QB8Ea69Pq9mlxLUdEnWM26fpv1ziUpWZcrSbbu8/lJSnFICndKxJr17MzcnW5+RLvNu2v92v9WwnKu/9L2ZJGKkqf3LFSk0E3N8CmKahOxmZvOtx4TskjZJ+qqkHZIuGbPdP5EUkjaW6nQHbGaLqz4FMWVCdkmrgMuAVwKnAOdLOiWz3VHA64Eb2zTPHbCZLbZ+ZsQ4HdgREXdFxF7gauCczHb/DngXB1Kej+UO2MwWV3/T0q8D7kme76zKniDpBcCGiPhk2+bN1UU4M7PetRvhrpW0PXm+JSK2tD2EpEOAdwOv69I0d8Bmtrja34q8OyLGXTTbBWxInq+vympHAc8DPisJ4Hhgq6SzIyLt2A8yeAec/nHKhU6Vwpm6zFe2umF53P5Nx8ppCkPLJbYpzRPXVO+4slKynqZtS/WWlN6X9D+8lXTOq6/opUnDrXJzAebmZCuFjpXWN31vlzOUcJaRYj2Fu90EnCzpJEYd73nAa+uVEfFdYG39XNJngd8b1/nCyvp9MDPrpKcgCCJiH3Ax8GngTuCaiLhd0jsknT1p+wYfAZuZzVJfo+uI2AZsW1L2loZtz2xTpztgM1tYc34nsjtgM1tsc3wnsjtgM1tc+5nrWenLHbCkK4BXAw9ExPOqsmOBDwEnAncD50bEg8W6gDXVcv3YNDVOySRXD6f9VyQ95rRX9ru81rbbpm3qMgnAtFMCzULTFfjclfuh2jKtttEjTVEQuSmHUm2jIErHh9l/x2dpnkfAbd7XDwCblpRdAlwXEScD11XPzczmSl9RELNS7IAj4gbgO0uKzwGurJavBF7Tc7vMzHoxzx3wpOeAj4uIe6vl+4DjemqPmVlv5nxGoukvwkVESIqm9ZI2A5t7OZiZWQdzPinyxHfC3S/pBIDq8YGmDSNiS0RsjIiN83JS3syePOb5FMSkHfBW4IJq+QLg4/00x8ysP/N+Ea5NGNoHgTMZpWvbCbwVeCdwjaQLga8D57Y94NJ5rJoSyOTmuyqFA+XCZpr26XJeqPRXapIPsEs4Ua3U5km/SH0mSpn1l7lUf59JY3IJoUrHym3blOymbZKlVBrTWgpTG1eW6uN3oe3vY5N6/1kkp1nR54Aj4vyGVWf13BYzs175VmQzs4G4AzYzG8i8R0G4AzazhbaizwGbma1UPgVhZjYgd8CV3G2B6fM0xGZVZn1JLsSnKfSsFK6TC6tJM7d1CVeqj5uG2DyWWd/nF6XLPHC5+enmVV//TjZltsvpkg2tSxhaTi6kbE9StiezbVMYWtv3qk2Wv1yYWO53qxRG1rS+Lu/7Zq2FvxXZzGyeeQRsZjYAR0GYmQ3EF+HMzAbkc8BmZgPwCLjB/iWPcPDV+Mcy69MrqLn5rpqiDNpqigbIRVSkSldwc/OY5a5cl5K+lK4Q596fNvt1kXstQyfgmXbusqa6Sm1pO7IqfRebEk7lohxKkTyptu/FqoblXMTDtOvbtKFv7oDNzAbgi3BmZgPxKQgzswH5IpyZ2QA8AjYzG4hvRTYzG5BHwImlIVddwplKf8mmDUcqhbyluswvl1vfFHqUq79tYp1J9pmVLuFas57HrIs+f1mnbWPdlvQq/sOZ+tMEPbn9uyiFka1u2JbC+lyoZu53fxbJeOY5CmI5fyfNzJZVn7MiS9ok6auSdki6JLP+jZLukHSrpOsk/XipTnfAZrbQ+uiAJa0CLgNeCZwCnC/plCWbfRHYGBH/APgw8B9K9boDNrOFVV+EK/20cDqwIyLuioi9wNXAOQcdK+L6iHikevp5YH2pUl+EM7OF1vIUw1pJ25PnWyJiS/J8HXBP8nwn8MIx9V0I/M/SQd0Bm9nC6hCGtjsiNvZxTEm/BmwEXlLadqoOWNLdwPcZ/ZHZ1+UFDBEaUkqWk8p9aF0SoZSOn1tuSl4y64iGXPRH05e2FLFQivgYVzbuuG3379Os40dLV/5zU12tyezfZXqlnKbvUimKIZe8qk1in9z6UlsmFRw81dkUdgEbkufrq7KDSHoZ8K+Bl0TEo6VK+xgB/+OI2N1DPWZmvevpD+lNwMmSTmLU8Z4HvDbdQNJpwJ8CmyLigTaV+hSEmS2svm5Fjoh9ki4GPs1o8H5FRNwu6R3A9ojYCvxH4EjgLyUBfCMizh5X77QdcACfkRTAny45aW1mNqg+c0FExDZg25KytyTLL+ta57Qd8IsjYpekHwOulfSViLgh3UDSZmAzzDbpsplZzjzngpjqnHdE7KoeHwA+xihWbuk2WyJiY0RsdAdsZsupvhW59DOUiTtgSUdIOqpeBl4B3NZXw8zMptXnrcizMM0piOOAj1Unmw8F/iIiPjVuB3EgdKUOpymFqnT567S/YXlcWdP6acPMckrhPqX1Xf6DmNUtjrMIMxsi8U7TnHnLOb9dKcys/l05PCnL/T6U3us+vgul7+jS7dqWL4eFzIYWEXcBz++xLWZmvXI+YDOzAS3kCNjMbN55SiIzs4HMe0J2d8BmttB8DtjMbAA+BZHI/TtwWLKcZleq37Quf70mfaPbhkG1yRqV0zYTVNP6Ura0XFlp2y6Z10pKoUnzPALpqjR/X2nbktxnmP7OpPO/1e/r3kxZk1KbunyXu8zp5jC0PI+AzWxhOQzNzGxAHgGbmQ1gP46CMDMbjEfAZmYD8DngRJqMp45+SCMfDsksNyXIySUymVZTUpa2EQddohAmvdo8zuMNy8s1t9mk6ydJ0LMc2rahz2Q+T0mWj8rUnybrqf+1bopCmTYioqRLJM40dU5rHr5LTTwCNrOF5ThgM7OB+FZkM7MBeQRsZjYAX4QzMxuQR8BmZgPwCDiRvhn1X6VSqErTCfRJ3tTSsbqsT8NlVi/dcInSfFptX0spxGdWYXCpXGKiacOduoxQlvOXqa+RU5c2p/O/1d+rpna0TazTtF3uMy61tel7kQvFbLPfUg5DMzNbEI6CMDMbiOOAzcwG4g7YzGxAvghnZjaAhR4BS9oEvIfRxcv3R8Q7x23/LODPquUzTq8Wfi7Z4CXJ8vHVY3r5dF2y/JRnVgvpJC2p3dVj+hL35Tf9wf4fruo7yXIuzOGRZPnh6jG9hJue+a/njEkzqTwtWX5q9Xh0UnboM5InP1k9Hp+Urc9UsDYpS5frgx2ZlK1pWK63Sd+r9I15iB+Wvsd1Xek+exq2za2vPZosr8qUN7XvyEzZoZnlpq9+7lhN37Fa+v6l7fpBpixdzrTls+cdWP6D0cPX/vZA0d8ke99RPX4zKUu/tnuWPEL+glRpmiE4eOqwnGmnpcpNv9SXeR4BT5KwCABJq4DLgFcCpwDnSzqlr4aZmU0rGI1/Sj9tSNok6auSdki6JLP+RyR9qFp/o6QTS3VO3AEDpwM7IuKuiNgLXA2cM0V9Zma9qu89KP2UtBxwXgg8GBHPAS4F3lWqd5oOeB1wT/J8JwefJDAzG9zjLX5aaDPgPAe4slr+MHCWJI2rdJoOuBVJmyVtl7Q9d/bQzGxW6otwPXTAbQacT2wTEfuA73Lw1Z4fMs1FuF3AhuT5+qrsIBGxBdgC8FwppjiemVlnLS/CrZW0PXm+peq7ZmqaDvgm4GRJJzHqeM8DXttLq8zMetDhVuTdEbFxzPo2A856m52SDmUUnvTtcQdVxOSDUkmvAv6EUfTKFRHx7wvbf4tR0NbucdutUGtZvNe1iK8J/LpWih+PiKdPU4GkT3FwTGaT3RGxaUw9hwJfA85i1NHeBLw2Im5PtrkI+OmI+B1J5wG/FBHnjm3fNB3wJCRtL/ylWZEW8XUt4msCvy6bTG7AKekdwPaI2CppDfDnwGmMQrLPi4i7xtXpO+HMzFqIiG3AtiVlb0mW9wD/tEudM4+CMDOzvCE64JlfWRzIIr6uRXxN4Ndlc2LZzwGbmdmIT0GYmQ3EHbCZ2UCWtQMuZRNaCSRtkHS9pDsk3S7p9VX5sZKulfR31eMxQ7d1EpJWSfqipE9Uz0+qMjvtqDI9lTITzhVJR0v6sKSvSLpT0s8swmcl6Xer799tkj4oac1K/6yejJatA16g9JX7gDdFxCnAGcBF1eu4BLguIk4Grquer0SvB+5Mnr8LuLTK8PQgo4xPK8l7gE9FxE8Bz2f02lb0ZyVpHfAvgY0R8TxGcannsfI/qyed5RwBL0T6yoi4NyK+UC1/n9Ev9DoOzoRNkC8BAAACIUlEQVR0JfCaYVo4OUnrgZ8H3l89F/BSRpmdYIW9LklPBX4WuBwgIvZGxEMswGfFKIb/R6s7tA4H7mUFf1ZPVsvZAS9c+soq4fJpwI3AcRFxb7XqPuC4gZo1jT8Bfp8D+UueBjxUZXaClfeZnQR8C/iz6rTK+yUdwQr/rCJiF/BHwDcYdbzfBW5mZX9WT0q+CDchSUcCHwHeEBHfS9fFKLZvRcX3SXo18EBE3Dx0W3p0KPAC4L0RcRqjPCQHnW5YoZ/VMYxG8ScBzwCOABrzGNj8Ws4OuFX6ypVA0mpGne9VEfHRqvh+SSdU608AHhiqfRN6EXC2pLsZnR56KaPzp0dX/+bCyvvMdgI7I+LG6vmHGXXIK/2zehnw9xHxrYh4DPgoo89vJX9WT0rL2QE/kb6yujp7HrB1GY/fi+q86OXAnRHx7mTVVuCCavkC4OPL3bZpRMQfRsT6iDiR0Wfz1xHxq8D1wC9Xm62o1xUR9wH3SKpnNT2L0VyWK/qzYnTq4QxJh1ffx/p1rdjP6slqWe+E65q+ch5JejHwv4Avc+Bc6ZsZnQe+Bngm8HXg3Ij4TraSOSfpTOD3IuLVkp7FaER8LPBF4Nci4tFx+88TSacyuqh4GHAX8JuMBh4r+rOS9HbgVxhF5XwR+C1G53xX7Gf1ZORbkc3MBuKLcGZmA3EHbGY2EHfAZmYDcQdsZjYQd8BmZgNxB2xmNhB3wGZmA/n/VFh7xdd6TbYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "rule = 'contextdm1'\n",
    "multitask.standard_analysis.easy_activity_plot(run_model, rule)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The RNN model can also be trained for other tasks by simply changing the `ruleset`.\n",
    "\n",
    "The basic RNN model worked well for this example, but in practice, many applications use a special type of RNN called a **long short-term memory (LSTM)** network. From now on, we'll call non-LSTM RNNs \"vanilla RNNs\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. LSTMs: Beyond vanilla RNNs\n",
    "\n",
    "To motivate the popularity of LSTMs, recall why we were interested in RNNs in the first place: modeling *sequences*. The idea is that items at later parts of a sequence can depend on items at earlier parts of the sequence. The distance between these related items is often called the **dependency length**.\n",
    "\n",
    "Dependency lengths can be quite short in some cases and quite long in others, even for the same task. Returning to the language example, suppose your task is to predict the last word in a sentence. If your sentence is \n",
    "\n",
    "> A car has four *wheels*.\n",
    "\n",
    "then the gap between the target word (here, \"wheels\") and the relevant parts of the sequence (here, \"car\") is relatively small. In contrast, consider the sentence \n",
    "\n",
    "> A car can have sentimental value for many owners for a variety of reasons, and can come in many models, sizes, and colors; nevertheless, one defining characteristic of such a machine is that it has four *wheels*.\n",
    "\n",
    "Here, the gap between \"wheels\" and \"car\" is relatively large. Words closer to the end of the sentence such as \"machine\" and \"four\" can help guide the prediction of \"wheels\", but we need the word \"car\" to nail down the correct word.\n",
    "\n",
    "In theory, vanilla RNNs are able to capture these dependencies of any length, but in practice, they often fail on long-term dependencies ([Bengio et al. 1994](http://ai.dinfo.unifi.it/paolo//ps/tnn-94-gradient.pdf)). LSTMs ([Hochreiter & Schmidhuber 1997](https://www.mitpressjournals.org/doi/abs/10.1162/neco.1997.9.8.1735)) turn out to be an effective solution to this problem.\n",
    "\n",
    "### 3.1 LSTM architecture\n",
    "\n",
    "Consider the following diagram of an LSTM (credit to [Shi Yan](https://medium.com/mlreview/understanding-lstm-and-its-diagrams-37e2f46f1714)):\n",
    "\n",
    "<img src =  \"img/lstm.png\"     style=\"text-align:center; width: 550px;\"/>\n",
    "\n",
    "\n",
    "There's a lot going on here, but in short, LSTMs have three multiplicative gates:\n",
    "\n",
    "1. The **input gate** controls what proportion of the input to pass to the memory cell.\n",
    "2. The **forget gate** controls what proportion of the previous memory cell information to discard.\n",
    "3. The **output gate** controls what proportion of the memory cell to output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Evaluating a pre-trained model\n",
    "\n",
    "Now that we've learned how to train a small RNN and have learned about LSTMs, let's examine the behavior of an LSTM that has already been pre-trained on a larger amount of data.\n",
    "\n",
    "In particular, we will be playing around with a **language model**. A language model assigns a probability distribution over words. It is trained to perform the following task: at each point in a sentence, predict the most probable next word given the preceding context. For example, if the first word of a sentence is \"The\", then a reasonable language model might assign high probabilities to nouns (e.g. \"dog\" or \"boy\") and low probabilities to verbs (e.g. \"swims\" or \"sings\") as the next word.\n",
    "\n",
    "How can we evaluate whether a language model has successfully \"learned\" English grammar, aside from using our intuition? One thing we can do is test the language model on some basic grammatical phenomena, such as **subject-verb agreement.**\n",
    "\n",
    "### 4.1 Subject-verb agreement\n",
    "\n",
    "In English, every well-formed sentence has a **subject** and a **verb**. For example, in the sentence \n",
    "\n",
    "> (1) The dog sniffs a bone.\n",
    "\n",
    "*dog* is the subject, and *sniffs* is the verb. The subject and verb must **agree** in **number** -- that is, a singular subject must be paired with a singular verb, and a plural subject must be paired with a plural verb. Returning to our example, this simply means that\n",
    "\n",
    "> (1) The dog sniffs a bone.\n",
    ">\n",
    "> (2) The dogs sniff a bone.\n",
    "  \n",
    "are both grammatically well-formed, but \n",
    "\n",
    "> (3) \\*The dog sniff a bone.\n",
    ">\n",
    "> (4) \\*The dogs sniffs a bone.\n",
    "  \n",
    "are not. (That's what the asterisk means!)\n",
    "\n",
    "### 4.2 Evaluation metric\n",
    "\n",
    "How does this relate to LSTM language models? The key assumption is that a language model that has successfully \"learned\" English should assign **higher probability** to grammatically well-formed sentences (\"grammatical\"), and **lower probability** to grammatically ill-formed sentences (\"ungrammatical\").\n",
    "\n",
    "This gives us a metric for evaluating our language model. Recall that language models assign probabilities at the level of each word. So a successful model should assign\n",
    "\n",
    "\\begin{align*}\n",
    "P(\\text{singular verb }|\\text{ singular subject}) &> P(\\text{plural verb }|\\text{ singular subject}) \\\\\n",
    "P(\\text{plural verb }|\\text{ plural subject}) &> P(\\text{singular verb }|\\text{ plural subject}).\n",
    "\\end{align*}\n",
    "\n",
    "Returning to our example, this means that we would expect\n",
    "\n",
    "\\begin{align*}\n",
    "P(\\text{sniffs }|\\text{ The dog}) &> P(\\text{sniff }|\\text{ The dog}) \\\\\n",
    "P(\\text{sniff }|\\text{ The dogs}) &> P(\\text{sniffs }|\\text{ The dogs}).\n",
    "\\end{align*}\n",
    "\n",
    "Finally, for reasons beyond the scope of this tutorial, psycholinguists are typically interested in the **negative log probability** of words, as [it is better correlated with human behavioral measures](https://www.mit.edu/~rplevy/papers/smith-levy-2013-cognition.pdf). Since this quantity is known as **surprisal**, let's denote it by $S$. This means that we will equivalently be looking for the pattern \n",
    "\n",
    "\\begin{align*}\n",
    "S(\\text{sniffs }|\\text{ The dog}) &< S(\\text{sniff }|\\text{ The dog}) \\\\\n",
    "S(\\text{sniff }|\\text{ The dogs}) &< S(\\text{sniffs }|\\text{ The dogs}).\n",
    "\\end{align*}\n",
    "\n",
    "So, does this actually hold in practice? Let's take a look below!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's import some helper functions from `utils.py` that we'll need for evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to download the parameters for the pre-trained model. This language model has been trained on an Americanized version of the [British National Corpus (BNC)](https://www.english-corpora.org/bnc/). You can **download the model checkpoint** [here](https://www.dropbox.com/s/er9exdbwun4rex9/model_bnc.pt?dl=1). Clicking on the link should download a file called `model_bnc.pt`, which you should move to the `materials` folder.\n",
    "\n",
    "Once you've done that, let's set some parameters and paths to important information: the model checkpoint, input sentences, and dictionaries containing vocabulary information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {\n",
    "    'checkpoint': 'materials/model_bnc.pt', # path to model checkpoint\n",
    "    'eval_data': 'materials/examples.txt', # path to eval data\n",
    "    'dicts': 'materials/lstm_bnc_dicts.json', # path to dictionaries\n",
    "    'temperature': 1.0, # higher temperature will increase diversity\n",
    "    'seed': 1111 # random seed for reproducibility\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we'll initialize some settings, e.g. random seed and device (CPU or GPU)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(params['seed'])\n",
    "device = torch.device(\"cpu\")\n",
    "assert(params['temperature'] >= 1e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load and initialize the model from the specified path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RNNModel(\n",
       "  (drop): Dropout(p=0.5)\n",
       "  (encoder): Embedding(68344, 400)\n",
       "  (rnn): LSTM(400, 400, num_layers=2, dropout=0.5)\n",
       "  (decoder): Linear(in_features=400, out_features=68344, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open(params['checkpoint'], 'rb') as f:\n",
    "    model = torch.load(f, map_location=lambda storage, loc: storage)\n",
    "    model.cpu()\n",
    "model.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the sentences and dictionaries from the specified paths. Note that the sentence file `params['eval_data']` is expected to have one sentence on each line, with tokens separated by spaces. (For the simple sentences that we'll be working with, just think of separating each word with a space, as well as the final period.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load eval sentences\n",
    "with open(params['eval_data'], 'r') as f:\n",
    "    lines = f.readlines()\n",
    "sents = [line.strip().split() for line in lines]\n",
    "\n",
    "# load the dictionaries, word2idx and idx2word, of the preprocessed corpus used by the model\n",
    "corpus_dict = utils.Dictionary()\n",
    "corpus_dict.load(params['dicts'])\n",
    "ntokens = len(corpus_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we need to preprocess the eval data in two ways. First, since there may be words in the eval data that the model has not seen during training, we need to map out-of-vocabulary words to fine-grained `UNK` tokens. Second, we add an `<eos>` token to signify the beginning of the sentence. Otherwise, the model would not be able to assign a probability to the first word, since there is no explicit preceding context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# map out-of-vocabulary words to \"UNK\" tokens\n",
    "sents = [utils.unkify(sent, corpus_dict.word2idx) for sent in sents]\n",
    "\n",
    "# add <eos> token to beginning of each sentence\n",
    "for sent in sents:\n",
    "    sent.insert(0, '<eos>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we're ready to perform the evaluation. Each word and its negative log probability will be printed on a new line, separated by a tab (`\\t`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SENTENCE: <eos> The dog sniffs a bone .\n",
      "<eos>\t--\n",
      "The\t2.4970164\n",
      "dog\t8.273961\n",
      "sniffs\t11.779587\n",
      "a\t2.979677\n",
      "bone\t7.6778884\n",
      ".\t2.6759293\n",
      "\n",
      "SENTENCE: <eos> The dogs sniff a bone .\n",
      "<eos>\t--\n",
      "The\t2.4970164\n",
      "dogs\t10.054913\n",
      "sniff\t11.877056\n",
      "a\t3.480141\n",
      "bone\t8.105723\n",
      ".\t2.6828048\n",
      "\n",
      "SENTENCE: <eos> The dog sniff a bone .\n",
      "<eos>\t--\n",
      "The\t2.4970164\n",
      "dog\t8.273961\n",
      "sniff\t13.05693\n",
      "a\t3.4009051\n",
      "bone\t7.726699\n",
      ".\t2.6417565\n",
      "\n",
      "SENTENCE: <eos> The dogs sniffs a bone .\n",
      "<eos>\t--\n",
      "The\t2.4970164\n",
      "dogs\t10.054913\n",
      "sniffs\t14.093539\n",
      "a\t3.5034008\n",
      "bone\t8.180921\n",
      ".\t2.7263448\n",
      "\n"
     ]
    }
   ],
   "source": [
    "input = torch.randint(ntokens, (1, 1), dtype=torch.long).to(device)\n",
    "\n",
    "with torch.no_grad():\n",
    "    for sent in sents:\n",
    "        # print the sentence\n",
    "        print('SENTENCE:', ' '.join(sent))\n",
    "        \n",
    "        # initialize the hidden layer\n",
    "        hidden = model.init_hidden(1)\n",
    "\n",
    "        # map the first word of the sentence to its index in the model vocabulary\n",
    "        input = torch.tensor([[corpus_dict.word2idx[sent[0]]]],dtype=torch.long).to(device)\n",
    "        print(sent[0]+'\\t'+'--')\n",
    "\n",
    "        for i, w in enumerate(sent[1:]):\n",
    "            output, hidden = model(input, hidden)\n",
    "            word_weights = torch.Tensor.numpy(output.squeeze().div(params['temperature']).exp().cpu())\n",
    "            total_weight = np.sum(word_weights)\n",
    "            word_idx = corpus_dict.word2idx[w]\n",
    "            word_surprisal = -np.log(word_weights[word_idx]/total_weight)\n",
    "            print(w+'\\t'+str(word_surprisal))\n",
    "            input.fill_(word_idx)\n",
    "        print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see \n",
    "\n",
    "\\begin{align*}\n",
    "S(\\text{sniffs }|\\text{ $<$eos$>$ The dog}) = 11.78 &< S(\\text{sniff }|\\text{ $<$eos$>$The dog}) = 13.05 \\;\\; \\checkmark \\\\\n",
    "S(\\text{sniff }|\\text{ $<$eos$>$ The dogs}) = 11.88 &< S(\\text{sniffs }|\\text{ $<$eos$>$The dogs}) = 14.09 \\;\\; \\checkmark\n",
    "\\end{align*}\n",
    "\n",
    "as expected. This suggests that the language model has indeed learned something about subject-verb number agreement!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4 Exploration\n",
    "\n",
    "Now, it's your turn to play around with the model by designing your own test sentences. You can edit `materials/examples.txt` directly and re-run all the cells after loading the eval data, or make your own file following the format of `materials/examples.txt` and update `params['eval_data']` accordingly. \n",
    "\n",
    "Do the results surprise you? What other grammatical phenomena would be good for assessing language models?\n",
    "\n",
    "(Note: if you're interested in training a language model yourself, see [this Pytorch example](https://github.com/pytorch/examples/tree/master/word_language_model).)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Useful resources\n",
    "\n",
    "### 5.1 Further reading\n",
    "\n",
    "[The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)\n",
    "\n",
    "[The Deep Learning textbook](http://www.deeplearningbook.org/) (Chapter 10 is most relevant)\n",
    "\n",
    "### 5.2 Further practice\n",
    "\n",
    "[Pytorch character generation tutorial](https://pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html)\n",
    "\n",
    "[Pytorch character classification tutorial](https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}