Continuous_Control.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continuous Control\n",
    "\n",
    "---\n",
    "\n",
    "You are welcome to use this coding environment to train your agent for the project.  Follow the instructions below to get started!\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "Run the next code cell to install a few packages.  This line will take a few minutes to run!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The environments corresponding to both versions of the environment are already saved in the Workspace and can be accessed at the file paths provided below.  \n",
    "\n",
    "Please select one of the two options below for loading the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "import torch\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "torch.cuda.is_available()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\tgoal_speed -> 1.0\n",
      "\t\tgoal_size -> 5.0\n",
      "Unity brain name: ReacherBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 33\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np\n",
    "\n",
    "# select this option to load version 1 (with a single agent) of the environment\n",
    "# env = UnityEnvironment(file_name='/data/Reacher_One_Linux_NoVis/Reacher_One_Linux_NoVis.x86_64')\n",
    "\n",
    "# select this option to load version 2 (with 20 agents) of the environment\n",
    "env = UnityEnvironment(file_name='/data/Reacher_Linux_NoVis/Reacher.x86_64')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 20\n",
      "Size of each action: 4\n",
      "There are 20 agents. Each observes a state with length: 33\n",
      "The state for the first agent looks like: [  0.00000000e+00  -4.00000000e+00   0.00000000e+00   1.00000000e+00\n",
      "  -0.00000000e+00  -0.00000000e+00  -4.37113883e-08   0.00000000e+00\n",
      "   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "   0.00000000e+00   0.00000000e+00  -1.00000000e+01   0.00000000e+00\n",
      "   1.00000000e+00  -0.00000000e+00  -0.00000000e+00  -4.37113883e-08\n",
      "   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "   0.00000000e+00   0.00000000e+00   5.75471878e+00  -1.00000000e+00\n",
      "   5.55726624e+00   0.00000000e+00   1.00000000e+00   0.00000000e+00\n",
      "  -1.68164849e-01]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents\n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.\n",
    "\n",
    "Note that **in this coding environment, you will not be able to watch the agents while they are training**, and you should set `train_mode=True` to restart the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env_info = env.reset(train_mode=True)[brain_name]      # reset the environment    \n",
    "states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "while True:\n",
    "    actions = np.random.randn(num_agents, action_size) # select an action (for each agent)\n",
    "    actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1\n",
    "    env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "    next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "    rewards = env_info.rewards                         # get reward (for each agent)\n",
    "    dones = env_info.local_done                        # see if episode finished\n",
    "    scores += env_info.rewards                         # update the score (for each agent)\n",
    "    states = next_states                               # roll over states to next time step\n",
    "    if np.any(dones):                                  # exit loop if episode finished\n",
    "        break\n",
    "print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. It's Your Turn!\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  A few **important notes**:\n",
    "- When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```\n",
    "- To structure your work, you're welcome to work directly in this Jupyter notebook, or you might like to start over with a new file!  You can see the list of files in the workspace by clicking on **_Jupyter_** in the top left corner of the notebook.\n",
    "- In this coding environment, you will not be able to watch the agents while they are training.  However, **_after training the agents_**, you can download the saved model weights to watch the agents on your own machine! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from DDPG_Agent import Agent\n",
    "\n",
    "agent = Agent(state_size=state_size, action_size=action_size, random_seed=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cuda', index=0)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agent.device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ddpg(n_episodes=1000, max_t=2000, print_every=10):\n",
    "    \"\"\"\n",
    "    DDPG \n",
    "    \n",
    "    Parameters\n",
    "    ==========\n",
    "        n_episodes (int): maximum number of training episodes\n",
    "        max_t (int): maximum number of timesteps per episode\n",
    "        \n",
    "    \"\"\"\n",
    "    # list containing scores from each episode\n",
    "    total_scores = []      \n",
    "    # last 100 scores\n",
    "    scores_deque = deque(maxlen=100)\n",
    "    \n",
    "    start_time = time.time()\n",
    "    \n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        # reset the environment\n",
    "        env_info = env.reset(train_mode=True)[brain_name]\n",
    "        # obtain the current state for each agent\n",
    "        states = env_info.vector_observations\n",
    "        # initialize the score (for each agent)\n",
    "        scores = np.zeros(num_agents)\n",
    "        agent.reset()\n",
    "                \n",
    "        for timestep in range(max_t):\n",
    "            # select an action\n",
    "            actions = agent.act(states, add_noise=True)\n",
    "            # send all actions to the environment\n",
    "            env_info = env.step(actions)[brain_name]        \n",
    "            # get the next states for each agent\n",
    "            next_states = env_info.vector_observations\n",
    "            # obtain the rewards for each agent\n",
    "            rewards = env_info.rewards\n",
    "            # see whether or not the episode is finished\n",
    "            dones = env_info.local_done\n",
    "            agent.step(states, actions, rewards, next_states, dones, timestep)\n",
    "            # roll over to the next timestep\n",
    "            states = next_states \n",
    "            # update the scores for each agent\n",
    "            scores += rewards\n",
    "            \n",
    "            if np.any(dones):\n",
    "                break \n",
    "                        \n",
    "        # average the scores of each agent\n",
    "        score = np.mean(scores)\n",
    "        # save most recent score        \n",
    "        scores_deque.append(score) \n",
    "        # save most recent score\n",
    "        total_scores.append(score)             \n",
    "        \n",
    "        \n",
    "        if i_episode % print_every == 0 or (len(scores_deque) == 100 and np.mean(scores_deque) >= 30) :\n",
    "            torch.save(agent.actor_local.state_dict(), 'checkpoint_actor.pth')\n",
    "            torch.save(agent.critic_local.state_dict(), 'checkpoint_critic.pth')\n",
    "            duration = (int)(time.time() - start_time) \n",
    "            print('Episode {}, AVG Score: {:.2f}'.format(i_episode, np.mean(scores_deque)))\n",
    "\n",
    "        \n",
    "        if len(scores_deque) == 100 and np.mean(scores_deque) >= 30:  \n",
    "            print('\\nEnvironment solved in {:d} episodes!\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_deque)))\n",
    "            torch.save(agent.actor_local.state_dict(), 'final_checkpoint_actor.pth')\n",
    "            torch.save(agent.critic_local.state_dict(), 'final_checkpoint_critic.pth')\n",
    "            break\n",
    "            \n",
    "            \n",
    "    return total_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Started.......\n",
      "/home/ubuntu/anaconda3/envs/drlnd/lib/python3.6/site-packages/torch/nn/functional.py:995: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.\n",
      "  warnings.warn(\"nn.functional.tanh is deprecated. Use torch.tanh instead.\")\n",
      "Episodes 10, AVG Score: 1.12\n",
      "Episodes 20, AVG Score: 1.41\n",
      "Episodes 30, AVG Score: 1.96\n",
      "Episodes 40, AVG Score: 3.22\n",
      "Episodes 50, AVG Score: 5.21\n",
      "Episodes 60, AVG Score: 7.42\n",
      "Episodes 70, AVG Score: 9.75\n",
      "Episodes 80, AVG Score: 12.15\n",
      "Episodes 90, AVG Score: 14.26\n",
      "Episodes 100, AVG Score: 16.41\n",
      "Episodes 110, AVG Score: 20.09\n",
      "Episodes 120, AVG Score: 23.75\n",
      "Episodes 130, AVG Score: 27.26\n",
      "Episodes 139, AVG Score: 30.03\n",
      "\n",
      "Environment solved in 139 episodes!\tAverage Score: 30.03\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "import time\n",
    "\n",
    "scores = ddpg()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = np.load('scores.npy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot the scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xd81fW9+PHXO3uTQRYkEJYiM2BkCFrFhaOOVotWK3WU2tZqx+926G2rvfbe29aK3mqt1oXW4t5aN4igAmFDWAECWZBBBtnjvH9/nG9igASi5ozkvJ+PRx455/P9fnPefMnJ+3y2qCrGGGMCV5CvAzDGGONblgiMMSbAWSIwxpgAZ4nAGGMCnCUCY4wJcJYIjDEmwFkiMMaYAGeJwBhjApwlAmOMCXAhvg6gNwYPHqxZWVm+DsMYY/qVNWvWVKhq8vHO6xeJICsri9zcXF+HYYwx/YqI7O3NeR5vGhKRYBFZJyJvOM9HiMhKEdkpIs+KSJinYzDGGNMzb/QR3Aps7fL8j8BCVR0DVAE3eCEGY4wxPfBoIhCRDOBC4BHnuQBzgBecUxYBl3oyBmOMMcfm6RrBvcAvAJfzPAmoVtU253kRMNTDMRhjjDkGjyUCEbkIKFPVNV2Luzm12w0RRGSBiOSKSG55eblHYjTGGOPZGsEs4GIRKQCewd0kdC8QLyIdo5UygJLuLlbVh1U1R1VzkpOPO/rJGGPMl+SxRKCqv1bVDFXNAq4EPlTVq4ElwOXOafOBVz0VgzHGmOPzxcziXwI/E5F83H0Gj/ogBmOM6bRuXxWf7KrwdRg+45UJZaq6FFjqPN4NTPPG6xpjzPEUVTVw7aOraGprZ/H3ZpCTlejrkLzO1hoyxvRbLpfy6vpi7nx9C9c9vootJTU9ntvS5uKz3ZXsKq+jtd09kLHdpfzsuQ0okD4okh88vZYDtU3UNbexq7zOS/8K3+sXS0wYY0x3nvy0gDtezyMiNIiQoCC+/9Qa3vjxbGLCQ1j4/g4q61q4ZsZwRODnz21g2/5DAIQECVOHJZAcG86qPQe5+4rJTMoYxKUPrODsez6irrkNVXjk2hzOHpfqk39bU2s7/95cyvkT0okIDfboa1kiMMb4TFu7i/mPr6K6oZVbzxrDOeNScc87dWtpcxEW0nPDxYtrixk/JI7Xbp7NpuIarvj7J9z6zHoUWLajnLCQIJ5ZXUiQQFJMOH+5YjIAO8oOsSK/gjc3HeTCiel8c+pQRIQHvj2Vf63ax/ghcby+oYTfv5HH7DGDPf6HuKuGljb++mE+z64u5GB9C6HBQVw0aYhHX9MSgTHGZx5cuosV+ZWkxUWw4Kk15AxP4G9XTyUlLoLncwu5/eXNfHdWFr+cO5bgoMOnIeWXHWJTcQ2/uWgcwUFCdmY8v/36eH7zymZCgoT/+cZELpiYzvO5hZTXNXPT6aNIiD58abOahlaiw4M7k8+ZY1M4c2wKAKdkJXL1Iyt55OPd3DxnDACqSn5ZHQfrW0iMDmNIfCTR4X37Z/TvS3fx4NJdnDc+lWtnZnHqqKQ+/fndsURgjPGJjUXV3PfBTi7JHsJfrpjMC2uK+P0beVx8/wouyR7CQ8t2k5EQycPLdrO7vJ6F8yYTGxHaef3L64oJEvj65PTOsmumDwNgbFospzidvjeeNrLHGAZFhfZ4bNbowZw/IY37l+RTUtNEcVUjm4prOFjf0nlObEQIT90wnezM+M4yVeX1jaVs319LsAjTRyYxa/TgXt0TVeWNTaWcOiqJh76T06tr+oKodjux16/k5OSoLUNtTP+xcncl8VFhnJgW21mmqqzac5BX1pdQXN3IluIawkKCePvW0zv/IOeV1HLjotWU1DRxwcQ0Fs7L5plVhdz5+hYSosL4wRmjuGbGcMKCgzjtT0sYnRLDous9NwixqKqBr/91OQBD4iMZmxbH9JGJDBkUSWV9M3e/u51DTW08//2ZjEmNpbmtnd++soVnc93NUS6F0GDh37eezuiUmMN+tsullNQ0UlDRwJjUGFLjIti2v5a5937MXZdO4JoZw79y/CKyRlWPm1EsERhj+kxFXTN3vp7H6xtKCAsO4jcXncTlJ2fyyvpiFn1SwLb9h4gND2FkcjSpcRH86MzRTO7yabrjZ3y8s5yLJw/tbA5at6+Ku9/dzor8ShKiQpk9JpnXN5Rw77xsLp3i2eXKVPWwfouu9lbWc/nfP8XlUsYNiaO0pon8sjp+PGc0Pz37BCrrWzjrL0sZNySOxd+b0flzmtvamXP3RxRXNwJwYmosb9wym79+mM/9H+5k5W1nkxwb/pVjt0RgjPGKwoMNXPfEaspqmzjU3EZoUBA3fW0kG4trWLq9nIjQIJpaXYxNi+W7p2ZxSfZQIsO+XOfrp7sq+edne3k3bz8RIcGsvP0sosJ828K9ff8h/vDWVg41tRIkwnWzsg7r3H165V5uf3kz93xrMt+YmgG4E9tlf/uEG2ePIG1QBHe9uZXbLhjLc7lFDI4J45kFM/sktt4mAusjMMZ8Jf9cuZc9FfVcM30Y8VFhfH1yOqNTYnG5lMdW7GH7/kNcfnIG00Yk9vjJurdmjkpi5qgkKuuaaWhp93kSADgxLZYnj9E8ddUpw3hhTRF/eHMrF05KJzwkmPWF1YC7/yI1LpxPd1Vy9zs7aGl3MX/meG+F3sn3d9EY02+1tLl4IbeIs8amcOclEw47FhQkx+yo/SqSYsLx/FiavhEUJPzwjNF878lc1u6tZuaoJDYUVpMaF07aoAgA7rh4POcs/AhxwXkT0rweoyUCY8yX9v7WA1TWt3CVM1rHdG/GyESCg4QV+RXMHJXE+sJqJmd83jeSmRjFf10ygT0V9aTERng9PltiwhjzpS1etY+h8ZGcPsaWij+W2IhQsjPjWZ5fQXVDCwWVDWQPO7yT/IqcTH4xd6xP4rNEYIzp5HIpFXXNvTo3v6yOj3dW8K2czKMme5mjzRqVxMaiapbtdK9ymp0Rf5wrvMcSgTGm06PL9zD9vz/gpbVF3R5vam3nzY2l3LhoNXPvXUZ4SBDfOiXDy1H2T7NGD8al8NBHuxCBiRmDfB1SJ+sjMMZ0envL/s4VOeua27h2Zlbnsfs/3MlDy3ZzqKmN1Lhwbpg9gityMkkfFOm7gPuRKcMSiAwNZktJLWNSYg6bJe1rlgiMMYB73Z11+6pYcPpIdpfX89tXt5AUHc6Fk9JZubuSu9/dwRknJnPD7BGcOmqwNQd9QWEhQUwfmcjS7eVHTaLzNWsaMsYAsGJXBS6Fc8el8uA1U5mcMYjbX9lESXUjv3ttC0PjI3nw6pM5bUyyJYEvabaz5lB2oCQCEYkQkVUiskFEtojInU75EyKyR0TWO1/ZnorBGNN7H20vJzYihOzMeEKDg7hnXjZNre1c8sAKtu0/xG8uOulLzwg2bnMnpDE5M54zTvSvUVaerBE0A3NUdTKQDcwVkRnOsf9Q1Wzna70HYzDG9IKqsmxnObNHDyYk2P1nYVRyDLdfcBLlh5qZPXow5433/kSngSYjIYpXfzSLjIQoX4dyGI/1Eah7EaOOvd5CnS//X9jImAC0s6yO0pombjnr8E+q18wYTkRoMF87IfkrLw9h/JdH+whEJFhE1gNlwHuqutI59AcR2SgiC0Wk2yX2RGSBiOSKSG55ebknwzQmIOyvaeK7j6/i012VRx1btsP9Hjv9hMMTgYhwRU4mKXHen+1qvMejiUBV21U1G8gAponIBODXwFjgFCAR+GUP1z6sqjmqmpOc7F/tacb0R5/sqmDp9nKufuQzHliSj8vlrqAXVzfyyMd7GJsWy9B4GwoaiLwyakhVq4GlwFxVLVW3ZuBxwHO7ShhjOpXWNAHuDss/v7Ody/62giXbyvjOoyupb2nj3itt3Eag8uSooWQRiXceRwJnA9tEJN0pE+BSYLOnYjDGfK6kupGEqFAe+PZUFs6bzP7aJq57YjVFVY08Ov8UxqbF+TpE4yOenFCWDiwSkWDcCec5VX1DRD4UkWRAgPXATR6MwRjjKK1pIn1QJCLCZVMyOHdcGk99tpeJQwcxbUSir8MzPuTJUUMbgSndlM/x1GsaY3pWUt1IRsLnfQDR4SHc9LVRPozI+AubWWxMgOioERhzJEsExgSAhpY2ahpbSY+3YaDmaJYIjAkAJdXuEUNDrEZgumGJwJgAUFrTCED6IKsRmKNZIjAmAJR21AhswpjphiUCYwJASU0jIpBqS0WYblgiMCYAlFY3MTgmnLAQe8ubo9lvhTEBoKSmkSHWP2B6YInAmAGmoKKeQ02th5WVVDfaHALTI0sExgwgB2qbOP++j7nqH5/R3NYOuDedKa1psjkEpkeWCIwZQO55dwct7S42F9fyx39vB6C2sY2GlnabQ2B65MlF54wxXuByKUFBwtbSWp5bU8gNs0bQ5lIeW7GHU0clMdRZX8hqBKYnlgiM6YdKqht5fMUePt5Zwc6yOqZlJXKouZW4iFBunjOayLBgVhcc5ObFa7ni5EwA6yMwPbKmIWP6ob8tzeexFQUkxYTxnRnDqahrZnNxLT85ewzxUWGEhwSz6PppnJgay1Of7QVgiNUITA+sRmBMP7RmbzWnjkriqRumd5aVH2pmcExY5/PBMeEsXjCDWxavZ83eg6TEWiIw3bNEYIwPqCp3vp7HmWNT+NoJX2xP7rrmNrbvr+WcOWMOK0+ODT/q3KiwEP5x7ck0tboIDpKvFLMZuKxpyBgf2FJSyxOfFPD717d0biLfWxsLq3EpTB0W36vzRYTIsOAvE6YJEJ7cszhCRFaJyAYR2SIidzrlI0RkpYjsFJFnRSTseD/LmIHmlXXFAOwqr2fJ9rIvdO3afVUATMlM6PO4TGDyZI2gGZijqpOBbGCuiMwA/ggsVNUxQBVwgwdjMMbvtLuU1zaUMGdsCkMGRfDQst20tbv45Qsbufj+5cetIazdV83olBgGRYV6KWIz0HksEahbnfM01PlSYA7wglO+CLjUUzEY448+3VVJ2aFmLj85g+tnj2DVnoNc9Y/PeDa3kI1FNawvqu7xWlVl3b6qXjcLGdMbHu0jEJFgEVkPlAHvAbuAalVtc04pAob2cO0CEckVkdzy8nJPhmmMV72yvpjY8BDmjE3hymnDiI0IYXVBFT89+wRCgoT38g4cdc2nuyopqKhnT0U9VQ2tTB1mzUKm73h01JCqtgPZIhIPvAyc1N1pPVz7MPAwQE5OzhfrTTPGTzW1tvP25v1cMDGNiFB3B+7/XTWFljYX541PY+WeSt7PO8Av547tvKastonvPLqSyNBgzp+YBsDJwy0RmL7jlVFDqloNLAVmAPEi0pGAMoASb8RgjD9Yu6+KuuY25k5I6yw788QUzhvvfn72SansLKujoKK+8/hzuYW0uZTkuHCeyy0iLiKEUckxXo/dDFyeHDWU7NQEEJFI4GxgK7AEuNw5bT7wqqdiMMbfbC6uAWByRvdt/OeMSwXg/a3u5qF2l7J4VSGzRifx6o9mcdGkdOadkkmQzQkwfciTTUPpwCIRCcadcJ5T1TdEJA94RkTuAtYBj3owBmP8yqbiWobGR5IUc/TkL4DMxCjGpsXybt4BbjxtJMt2lFNc3cjtF55EbEQo9397qpcjNoHAY4lAVTcCU7op3w1M89TrGuPPNhVVM2Fo3DHPOWdcKg8syeeBJfl8squC5NjwzpqCMZ5gM4uN8ZLaplYKKhuY1EOzUIdrZgwnJyuRP7+znRX5lczLySQ02N6qxnNsrSFj+tC2/bWkD4pkUOTRk706+gcmDB10zJ+RGhfBc9+fyd7Kej7aUc6lU7odYW1Mn7GPGcb0kS0lNVz4f8u59IEV7K10j/opqKjvTAAd3yceJxF0GJ4UzbUzs4iLsBnExrOsRmBMH2h3Kbe9vJn4yFCqG1r4xt8+YXJmPEu2lxEaFMR7PzudjUU1DI2PJDHaltcy/sVqBMb0gX+t3MuGwmp++/VxvPiDU4kOD2FDYTU3fW0UIcHCXW9uZXNxTa9rA8Z4k9UIjPmKqupb+NPb25k1OomLJw9BRHjvZ6cjCGEhQcRFhPLHt7cBcEVOpo+jNeZoViMw5itanl/BoeY2fn7uiYi4J3qFhwQTFuJ+e10/O4uspCjg+B3FxviCJQJjvqLVBQeJCgtmUg9/5MNDgvnDZROZOHSQrRpq/JI1DRnzFa0uqGLqsARCjjHWf9bowbz+49lejMqY3rMagTFfQW1TK9v215KTZauBmv7LEoExX8GavVWowrSsRF+HYsyXZonAmK8gt+AgwUFCtrX9m37MEoExX8HqgiomDIkjKsy620z/ZYnAmC+pua2dDYXVnGLNQqafs48xxnxBD320i5fXFZOREEVzm4scSwSmn7MagTFfwObiGv70znZa2lys3VdFdFgw00ZYIjD9m8dqBCKSCTwJpAEu4GFVvU9E7gC+B5Q7p96mqm95Kg5j+kpru4tfvLCRxOgwXv7hLOIiQ2ht184ZxMb0V55sGmoDfq6qa0UkFlgjIu85xxaq6t0efG1j+tzDy3aTV1rL3685mUFR7qWhw0Js72DT/3lyq8pSoNR5fEhEtgK2w4bpl9raXTzy8W7OGpvC3Alpvg7HmD7llTqtiGTh3r94pVN0s4hsFJHHRMSmZBq/t7qgiqqGVi4/OcPXoRjT5zyeCEQkBngR+Imq1gIPAqOAbNw1hr/0cN0CEckVkdzy8vLuTjHGa97Zsp/wkCC+dmKyr0Mxps95NBGISCjuJPC0qr4EoKoHVLVdVV3AP4Bp3V2rqg+rao6q5iQn25vP+I6q8u6W/Zw2JtkmjpkByWOJQNwLsz8KbFXVe7qUp3c57TJgs6diMOaL2ra/lp89u56yQ02dZZuKayipabK+ATNgefLjzSzgO8AmEVnvlN0GXCUi2YACBcD3PRiDMV/I05/t46V1xazdV8U/b5xORkIUb2/eT3CQcPZJKb4OzxiP8OSooeVAd2PrbM6A8Vsr91QyOiWGA7VNfPPBT5gxMolPd1UyfUQi8VG26bwZmGwmjDGOyrpmdhyo47IpQ3lmwQyGJUaxvrAaBa6ePtzX4RnjMdbzZYxj1Z6DAMwYmcT4IYN4/qZTfRyRMd5hNQJjHCv3HCQyNJhJGbbBvAkslgiMcXy2u5KThycQeoy9h40ZiOw33higqr6FbfsPMd1WEjUByBKBMcCqAnf/wPSRST6OxBjvs0RgDLBy90HCQ4KYnGn9AybwWCIwAaelzcWjy/dQWtPYWbZyTyVThyUQHhLsw8iM8Q1LBCag1DW3cf0Tq/mvN/J4cOkuAGoaWskrrWX6SOsfMIGp14lARGaLyHXO42QRGeG5sIzpe3XNbVz18Gd8uruSzMRIPtxWhqqyuuAgqjB9hPUPmMDUq0QgIr8Dfgn82ikKBf7pqaCM8YTnVheyqbiGB6+eyvdPH0VRVSO7yutZuaeSsJAgpgyL93WIxvhEb2cWX4Z7Y5m1AKpa4mw/aUy/oKo8l1vIpIxBnDs+jaKqBgCWbi9j5Z6DZGfGExFq/QMmMPW2aahFVRX3iqGISLTnQjKm720qrmHb/kN8KycTgIyEKMakxPD6xlI2F9cww+YPmADW20TwnIg8BMSLyPeA93FvKmNMv/Ds6kLCQ4K4OHtIZ9mZY1PYUFiNS23+gAlsvUoEqno38ALu3cZOBH6rqn/1ZGDG9JXGlnZeW1/CBRPTiYsI7Sw/w9l2MjRYmDrMts42geu4fQQiEgy8o6pnA+95PiRj+ta7efs51NzW2SzUIWd4IjHhIYxNiyUyzPoHTOA6biJQ1XYRaRCRQapa442gjOlLy3ZUkBgddtQ6QmEhQdx9xSSSY8N9FJkx/qG3o4aacG85+R5Q31Goqrf0dIGIZAJPAmmAC3hYVe8TkUTgWSAL91aV31LVqi8VvTG9sLrgIKdkJRAUdPSGeXMnpHdzhTGBpbeJ4E3n64toA36uqmudoaZrnETyXeADVf1fEfkV8CvccxSM6XOlNY3sO9jA/FOzfB2KMX6rV4lAVReJSBhwglO0XVVbj3NNKVDqPD4kIluBocAlwBnOaYuApVgiMB7SsevYtCwbHmpMT3qVCETkDNx/tAtwb0ifKSLzVXVZL6/Pwj0hbSWQ6iQJVLVURFK+cNTG9NLqgoPEhIdwUrrNfzSmJ71tGvoLcK6qbgcQkROAxcDJx7tQRGJwDzv9iarWihzdTtvDdQuABQDDhg3rZZjGwItrihgSH8nMUUms2nOQk4cnEGK7jhnTo96+O0I7kgCAqu7Avd7QMYlIKO4k8LSqvuQUHxCRdOd4OlDW3bWq+rCq5qhqTnJyci/DNIFuV3kd//HCBm5ctJo1ew+y40Ad02zWsDHH1NtEkCsij4rIGc7XP4A1x7pA3B/9HwW2quo9XQ69Bsx3Hs8HXv2iQRvTk/s/zCcsJIiwkCDmP7YawBKBMcfR20TwA2ALcAtwK5AH3HSca2YB3wHmiMh65+sC4H+Bc0RkJ3CO89yYr2x3eR2vri/mOzOGc8+8bOqa2wgLCWJShu06Zsyx9LaPIAS4r+OTvTPb+JizcFR1Oe6O5e6c1esIjeml+5e4awMLTh9Fcmw4/3nhSVTWt9iuY8YcR28TwQfA2UCd8zwSeBc41RNBGXM8pTWNbCqq4dzxaQCUVDfy6voSvntqVudM4RtPG+nLEI3pN3rbNBShqh1JAOdxlGdCMubYaptaufqRlSx4ag2bityrnjyXW0i7S/muTRwz5gvrbSKoF5GpHU9EJAdoPMb5xnhEu0u5dfE69lU2EBkazCPLd9PuUp5bXchpYwaTmWifT4z5onrbNPQT4HkRKcG9Oc0QYJ7HojIG965iS3eUMy0rkehw96/qg0vzWbK9nLsuncCeinqe+KSA6SOSKKlp4vYLx/k4YmP6p2PWCETkFBFJU9XVwFjci8W1AW8De7wQnwlgq/Yc5LrHV/PM6sLOsre37GfaiESumTGc62ZlAfC71zaTGB3G2eNskroxX8bxmoYeAlqcxzOB24AHgCrgYQ/GZQyPrXB/1thS7O4HaGt3seNAHdmZ7k3mMxKiOH9CGq3tyjenDrXRQcZ8ScdrGgpW1YPO43m4l5J+EXhRRNZ7NjQTyAoPNvBu3gEA8kprAdhdUU9Lm+uwdYNunjOanQfq+M6MLF+EacyAcLwaQbCIdCSLs4APuxzrbf+CMV/Yok8KCBbhm1MzyC+ro6m1na1OQjgpPa7zvLFpcbzz09MZlmSdxMZ8WcdLBIuBj0TkVdyjhD4GEJHRgO1WZjyirrmNZ1cXcsHEdOaMTaHNpew8UEdeaS2hwcKo5Bhfh2jMgHLMT/Wq+gcR+QBIB95VVXUOBQE/9nRwJjC9vdm9x/D8U7NIig4DIK+0hq2lhxidEkuorSRqTJ/qzZ7Fn3VTtsMz4RgDS7aXkRIbztRh8ahCdFgweSW1bC2t5bQxg30dnjEDjn20Mn6lrd3FxzvK+doJyYgIQUHCSelxfLyzgvJDzYzr0j9gjOkblgiMX1lXWE1tUxtnjv18TsD4IXHsrqgHDu8oNsb0DUsExq8s2VZGcJAwu0sT0Lghn//xt0RgTN+zRGB85sNtB9jjfNLvsHR7OScPTyAu4vMN8Malu/cTSI0LJ9HpPDbG9B1LBMYnWtpc3PTPtfzhza2dZftrmsgrreXMEw9fKmJMagwhTl+BMabv2aQw4xM7Dhyipc3F8vxyGlvaiQwL5qMd7u2rzxx7+B7VEaHB/PDM0UwcajuNGeMJHqsRiMhjIlImIpu7lN0hIsVHbF1pAtCGomoAmlpdLNtZDsALa4oYnhTFiamxR53/s3NO4JxxqV6N0ZhA4cmmoSeAud2UL1TVbOfrLQ++vvFjm4pqGBQZSlxECO/lHWBraS2rC6q4ZvpwRHra4dQY4wkeaxpS1WUikuWpn2/6t41FNUzKGERSdBgfbD1AsAjhIUFckZPh69CMCTi+6Cy+WUQ2Ok1HCT2dJCILRCRXRHLLy8u9GZ/xsKbWdrYfOMTkjHjOHZ9GVUMrz60p5OLJQ4iPslFBxnibtxPBg8AoIBsoBf7S04mq+rCq5qhqTnJyck+nmX5ic3ENt7+8idqmVvJKa2l3KRMzBnH6CcmEBQehCtfOzPJ1mMYEJK+OGlLVAx2PReQfwBvefH3jG5/uquR7T+ZS19xGbEQo6YMiAJiUMYiY8BDOGZ9KZV0zEzNsVJAxvuDVRCAi6apa6jy9DNh8rPNN//fZ7krmP76K4YlRDE+K5rEVezh5WALJseGkxbkTwv9dOYXPF7Y1xnibxxKBiCwGzgAGi0gR8DvgDBHJBhQoAL7vqdc3/uGltUVEhQXz/E0zqW9p58y7l/Lp7krOGpvSOTooOEgAGylkjK94ctTQVd0UP+qp1zP+qaCigTEpMcRHhREfBTfMHsGDS3dZM5AxfsRmFhuP2lNZzxknfN7Z/4MzRlFU1chFk9J9GJUxpitLBMZj6prbKD/UTNbg6M6yuIhQ/nrVFB9GZYw5ki06Z/rU6xtK2FVeB0CBs7LoiC6JwBjjfywRmD7T1u7ip8+u5/4P8wEoqHQngqwkSwTG+DNLBKbPlNY00eZS1he6F5TrqBFkDY7yZVjGmOOwRGD6TFFVIwB7Kuqpqm9hT0UDqXHhRIVZV5Qx/swSgekzhVUNnY/XF1VTUFlvzULG9AOWCEyfKTrYQJBAkMC6fdUUVNRbR7Ex/YDV2U2fKaxqJH1QJLERISzbUU5lfcthQ0eNMf7JagSmzxQebCAjIZIpwxI6O4ytacgY/2eJwPSZoqpGMhOjmDIsvrPMmoaM8X+WCEyfaG5r58ChJjISIpnaJREMT7Kho8b4O+sjMH2iuKoRVchMiGLk4BhiI0KIDQ8hIjTY16EZY47DEoHpE4XOHILMxCiCgoQ5Y1N8HJExprcsEZg+UeTMIchMjATg3nnZnfsNGGP8m/URmD5ReLCR0GAhJda965glAWP6D48lAhF5TETKRGRzl7JEEXlPRHY63xM89frGuwqrGhgaH+nsNmaM6U88WSN4Aph7RNmvgA9UdQzwgfPcDABFBxvITLQRQsb0Rx5LBKq6DDh4RPElwCLn8SLgUk+9vvGuoqpGMhIfg26FAAAQvUlEQVQifR2GMeZL8HYfQaqqlgI4321oyQBQ39xGZX0LGQlWIzCmP/LbzmIRWSAiuSKSW15e7utwzDHscfYdsMljxvRP3k4EB0QkHcD5XtbTiar6sKrmqGpOcnJyT6cZP9CxrtDkjPjjnGmM8UfeTgSvAfOdx/OBV738+sYD1u2rZnBMmPURGNNPeXL46GLgU+BEESkSkRuA/wXOEZGdwDnOc9PPrSusIjszweYOGNNPeWxmsape1cOhszz1mqbvNba0880HP2HeKZnMPzXrqOPVDS3sLq/nm1MzvB+cMaZP+G1nsfEPT35aQF5pLXe/u53qhpajjnf0D3RdetoY079YIjA9qm1q5cGPdjEuPY665jYeXLrrqHPW7asmSGCSdRQb029ZIjA9euTjPVQ3tPKnyydx2ZShPPFJAaU1jYeds66wmhNSY4kJt/ULjemvLBGYblU3tPDox7u5cGI6E4YO4qdnn4BLlV+8sJEDtU0AuFzK+n1VTBlmS0YZ05/ZxzjTrY92lFPf0s6Np40A3PsM/Oaicdz1xlbm3L2Ua0/NIik6jNqmNusfMKafs0RgurUiv4JBkaGHtf1fOzOL08ckc9ebeYf1F5ySleiLEI0xfcQSgTmKqrJ8ZwWnjko6alnprMHRPDL/FBpb2imubqS13WUb1BvTz1kiMEcpqGygpKaJH545uMdzIsOCGZ0S48WojDGeYp3F5ijLd7oX+TttTM+JwBgzcFgiMEdZnl9BRkIkw2yjGWMCgiUCc5h2l/LJrkpmjx5sawcZEyAsEZjDbCyq5lBTG7NGW7OQMYHCEoHp1NTazp2v5xEZGmyJwJgAYokgQLlciqp2PldVbntpE+sLq1k4bzKJ0WE+jM4Y4002fDQArciv4NZn1lHf3M6Q+AhiwkNoanWx/cAhfnr2CcydkO7rEI0xXmSJIMAs+qSA37+Rx8jB0VySnUxxVSONre0EBwnnTUjjlrNG+zpEY4yXWSIIII8u38N/vZHH2SelsHBeNrERob4OyRjjB3ySCESkADgEtANtqprjizgCyXt5B7jrzTzOG5/K364++ailI4wxgcuXNYIzVbXCh68fMD7bXckti9cxaegg7p03xZKAMeYw1jQ0gLW7lL8tyWfh+zvISormH/NziAwL9nVYxhg/46vhowq8KyJrRGSBj2IY8P709jb+8t4OLp48hNd+PJuU2Ahfh2SM8UO+qhHMUtUSEUkB3hORbaq6rOsJToJYADBs2DBfxNivNbe188zqQi6clM7Cedm2XIQxpkc+qRGoaonzvQx4GZjWzTkPq2qOquYkJyd7O8R+b8m2MmoaW5mXk2lJwBhzTF5PBCISLSKxHY+Bc4HN3o5joHtxbTEpseG2VIQx5rh80TSUCrzsfEoNAf6lqm/7II4B62B9C0u2lXH97BE2QsgYc1xeTwSquhuY7O3XDSSvbyihzaV8Y+pQX4dijOkHbNG5AUZVeS63kHHpcYxNi/N1OMaYfsASwQDz6a5KtpTUcvUMG2lljOkdSwQDzIMf7WJwTDjfnJrh61CMMf2EJQI/V1bbhMulR5UXVNSz8L0d1DS0dpZtKanh450VXDcri4hQm0FsjOkdSwR+rLi6kdl/WsItz6w7bBMZgNte3sR9H+zknIUf8fbm/eSV1HLf+zuJCQ/hmhnDfRSxMaY/srWG/NhLa4poaXPxxsZSRiXH8NNzTgBg+c4KPtlVybUzh/PZ7kpu+ueazmt+cMYoBkXa8tLGmN6zROBHFq/ax79W7uOpG6YxKDKUF9YWMWNkIpkJUdz3wU5iI0K4dmYWf3x7G0PjI7n9wpMAeD+vjCCBhOgwpmUl+vhfYYzpbywR+IktJTX87tUttLS7+PM727l0ylD2VjZwy5wxfH3yECrqmrnrza08tGw35Yea+fPlkwgPcfcDXDjJtpY0xnx5lgi8rKq+hVueWcflJ2dwSbZ7wldjSzu3PrOe+KhQTj8hmX+t2sfW0lqiwoKZOyGNsJAgHvvuKXy4rYw/v7Od9EERfMNGBRlj+oglAi/777e28vHOCpbnV3CoqY0pw+L5n7e2kV9Wx1M3TCM7M56Pd5azdl81l5+cQXS4+79IRDjrpFTOOikVVbWF5IwxfcYSQR/JL6sjIyHymMM2P8mv4Pk1RVw/awR7K+v5z1fca+3FRoTw+0vGc9oY9yqrd3x9PD/611qumpbZ7c+xJGCM6UuWCHqpqbWd/LI6iqoaqW5oITE6jHPGpSIivLimiJ8/v4Gk6DCunj6MoCBhRX4FQ+Mj+cNlE4kOD6GhpY3bXt7E8KQofjH3RIKDhD/+exuxEaF8d1bWYSN9zp+Yzrrfnmujf4wxXmGJoAdNre089ele1hdVs620lj0V9Rw5r+viyUP4+uQh/PLFjUzLSiQuMoS/LskHYFx6HK9tKGF3RT0/njOGP7yZx96DDTx1/fTOWsN/XjSux9e3JGCM8RY5cqKSP8rJydHc3Fyvvd7+mia+/881bCisZlhiFGPTYhmbFsuJaXEMT4oiITqMV9YV85d3t+NSODE1lud/MJO4iFD21zQRHhJEQnQY7+cd4ObFa2lqdZGREMmfLp/EqaNsfwBjjHeIyBpVzTnueYGQCJ76tIB73ttBeEgwCdFhZGcOYsbIJM46KZUYpzN2a2ktr28o4UBtMx/tKKexpY175mVz3vi0Hn/u8p0VPL1yL7+5aBxD4iO7PWdDYTUf7yznulkjOjt+jTHGGywROB5etov/fmsb00ckMjwpigO1zazdV8WhpjbS4iK44+LxlNY08j9vbcOlSkpsOFmDo7nj4vGckBrbx/8SY4zxnt4mggH9EfWBJfn8+Z3tXORs4B4a7F5aqd2lrNpzkDtf39K5PMNZY1P48xWTSYwO82XIxhjjdT5JBCIyF7gPCAYeUdX/9cTrjBgczbdyMvifb0w6bMvG4CBh5qgkXv/xbJ7+bC+hIUF8e9owG5ZpjAlIXm8aEpFgYAdwDlAErAauUtW8nq7xdmexMcYMBL1tGvLFMtTTgHxV3a2qLcAzwCU+iMMYYwy+SQRDgcIuz4ucMmOMMT7gi0TQXUP8Ue1TIrJARHJFJLe8vNwLYRljTGDyRSIoArouopMBlBx5kqo+rKo5qpqTnJzsteCMMSbQ+CIRrAbGiMgIEQkDrgRe80Ecxhhj8MHwUVVtE5GbgXdwDx99TFW3eDsOY4wxbj6ZR6CqbwFv+eK1jTHGHM4XTUPGGGP8SL9Ya0hEyoG9X/LywUBFH4bjaRav5/W3mC1ezxrI8Q5X1eOOtukXieCrEJHc3sys8xcWr+f1t5gtXs+yeK1pyBhjAp4lAmOMCXCBkAge9nUAX5DF63n9LWaL17MCPt4B30dgjDHm2AKhRmCMMeYYBnQiEJG5IrJdRPJF5Fe+judIIpIpIktEZKuIbBGRW53yRBF5T0R2Ot8TfB1rVyISLCLrROQN5/kIEVnpxPuss3SIXxCReBF5QUS2Ofd5pj/fXxH5qfO7sFlEFotIhD/dXxF5TETKRGRzl7Ju76e4/Z/z/tsoIlP9JN4/O78PG0XkZRGJ73Ls106820XkPG/H21PMXY79PxFRERnsPO+TezxgE4GzAc4DwPnAOOAqERnn26iO0gb8XFVPAmYAP3Ji/BXwgaqOAT5wnvuTW4GtXZ7/EVjoxFsF3OCTqLp3H/C2qo4FJuOO2y/vr4gMBW4BclR1Au4lWK7Ev+7vE8DcI8p6up/nA2OcrwXAg16KsasnODre94AJqjoJ9yZZvwZw3ntXAuOda/7m/B3xtic4OmZEJBP3hl77uhT3yT0esImAfrABjqqWqupa5/Eh3H+khuKOc5Fz2iLgUt9EeDQRyQAuBB5xngswB3jBOcVv4hWROOB04FEAVW1R1Wr8+P7iXvYlUkRCgCigFD+6v6q6DDh4RHFP9/MS4El1+wyIF5F070Tq1l28qvquqrY5Tz/DvQIyuON9RlWbVXUPkI/774hX9XCPARYCv+DwZfv75B4P5ETQrzbAEZEsYAqwEkhV1VJwJwsgxXeRHeVe3L+MLud5ElDd5Y3lT/d5JFAOPO40ZT0iItH46f1V1WLgbtyf+EqBGmAN/nt/O/R0P/vDe/B64N/OY7+NV0QuBopVdcMRh/ok5oGcCHq1AY4/EJEY4EXgJ6pa6+t4eiIiFwFlqrqma3E3p/rLfQ4BpgIPquoUoB4/aQbqjtO2fgkwAhgCROOu+h/JX+7v8fjz7wYicjvu5tmnO4q6Oc3n8YpIFHA78NvuDndT9oVjHsiJoFcb4PiaiITiTgJPq+pLTvGBjuqd873MV/EdYRZwsYgU4G5qm4O7hhDvNGWAf93nIqBIVVc6z1/AnRj89f6eDexR1XJVbQVeAk7Ff+9vh57up9++B0VkPnARcLV+PobeX+MdhfvDwQbnvZcBrBWRNPoo5oGcCPx+Axynff1RYKuq3tPl0GvAfOfxfOBVb8fWHVX9tapmqGoW7vv5oapeDSwBLndO86d49wOFInKiU3QWkIef3l/cTUIzRCTK+d3oiNcv728XPd3P14BrnZEtM4CajiYkXxKRucAvgYtVtaHLodeAK0UkXERG4O6AXeWLGLtS1U2qmqKqWc57rwiY6vx+9809VtUB+wVcgHtUwC7gdl/H0018s3FX4zYC652vC3C3u38A7HS+J/o61m5iPwN4w3k8EvcbJh94Hgj3dXxd4swGcp17/AqQ4M/3F7gT2AZsBp4Cwv3p/gKLcfdftDp/kG7o6X7ibrZ4wHn/bcI9Gsof4s3H3a7e8Z77e5fzb3fi3Q6c7y/3+IjjBcDgvrzHNrPYGGMC3EBuGjLGGNMLlgiMMSbAWSIwxpgAZ4nAGGMCnCUCY4wJcJYIzIAmIu0isr7L1zFnFovITSJybR+8bkHHCpFf8LrzROQOEUkQkbe+ahzG9EbI8U8xpl9rVNXs3p6sqn/3ZDC9cBruCWSnAyt8HIsJEJYITEBypuo/C5zpFH1bVfNF5A6gTlXvFpFbgJtwr0eTp6pXikgi8BjuSV4NwAJV3SgiSbgnAiXjnvwlXV7rGtzLS4fhXlTwh6rafkQ883AvhzwS93pDqUCtiExX1Ys9cQ+M6WBNQ2agizyiaWhel2O1qjoNuB/3mklH+hUwRd3r1t/klN0JrHPKbgOedMp/ByxX9+J2rwHDAETkJGAeMMupmbQDVx/5Qqr6LO51kDar6kTcM4unWBIw3mA1AjPQHatpaHGX7wu7Ob4ReFpEXsG9PAW4lwX5JoCqfigiSSIyCHdTzjec8jdFpMo5/yzgZGC1e/kgIul5kbsxuJcKAIhS9x4VxnicJQITyLSHxx0uxP0H/mLgNyIynmMv+9vdzxBgkar++liBiEguMBgIEZE8IF1E1gM/VtWPj/3PMOarsaYhE8jmdfn+adcDIhIEZKrqEtwb8cQDMcAynKYdETkDqFD3HhJdy8/HvbgduBdhu1xEUpxjiSIy/MhAVDUHeBN3/8CfcC+SmG1JwHiD1QjMQBfpfLLu8LaqdgwhDReRlbg/EF11xHXBwD+dZh/BvWdwtdOZ/LiIbMTdWdyx/PKdwGIRWQt8hLOvrKrmich/Au86yaUV+BGwt5tYp+LuVP4hcE83x43xCFt91AQkZ9RQjqpW+DoWY3zNmoaMMSbAWY3AGGMCnNUIjDEmwFkiMMaYAGeJwBhjApwlAmOMCXCWCIwxJsBZIjDGmAD3/wGk0gjoyJ0naAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f476d227b38>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot the scores\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(len(scores)), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.savefig('plots/dqn.png')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Watch a smart agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent.qnetwork_local.load_state_dict(torch.load('final_checkpoint.pth'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env_info = env.reset(train_mode=False)[brain_name]     # reset the environment    \n",
    "states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "while True:\n",
    "    actions = np.random.randn(num_agents, action_size) # select an action (for each agent)\n",
    "    actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1\n",
    "    env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "    next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "    rewards = env_info.rewards                         # get reward (for each agent)\n",
    "    dones = env_info.local_done                        # see if episode finished\n",
    "    scores += env_info.rewards                         # update the score (for each agent)\n",
    "    states = next_states                               # roll over states to next time step\n",
    "    if np.any(dones):                                  # exit loop if episode finished\n",
    "        break\n",
    "print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}