pratical-rl/week3_model_free/prioritized_experience_replay.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Honor Track: experience replay\n",
    "_This notebook builds upon `qlearning.ipynb`, or to be exact, generating qlearning.py._\n",
    "\n",
    "There's a powerful technique that you can use to improve sample efficiency for off-policy algorithms: [spoiler] Experience replay :)\n",
    "\n",
    "The catch is that you can train Q-learning and EV-SARSA on `<s,a,r,s'>` tuples even if they aren't sampled under current agent's policy. So here's what we're gonna do:\n",
    "\n",
    "<img src=https://github.com/yandexdataschool/Practical_RL/raw/master/yet_another_week/_resource/exp_replay.png width=480>\n",
    "\n",
    "#### Training with experience replay\n",
    "1. Play game, sample `<s,a,r,s'>`.\n",
    "2. Update q-values based on `<s,a,r,s'>`.\n",
    "3. Store `<s,a,r,s'>` transition in a buffer. \n",
    " 3. If buffer is full, delete earliest data.\n",
    "4. Sample K such transitions from that buffer and update q-values based on them.\n",
    "\n",
    "\n",
    "To enable such training, first we must implement a memory structure that would act like such a buffer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "from IPython.display import clear_output\n",
    "\n",
    "#XVFB will be launched if you run on a server\n",
    "import os\n",
    "if type(os.environ.get(\"DISPLAY\")) is not str or len(os.environ.get(\"DISPLAY\"))==0:\n",
    "    !bash ../xvfb start\n",
    "    %env DISPLAY=:1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "from collections import deque\n",
    "\n",
    "class ReplayBuffer(object):\n",
    "    def __init__(self, size):\n",
    "        \"\"\"\n",
    "        Create Replay buffer.\n",
    "        Parameters\n",
    "        ----------\n",
    "        size: int\n",
    "            Max number of transitions to store in the buffer. When the buffer\n",
    "            overflows the old memories are dropped.\n",
    "            \n",
    "        Note: for this assignment you can pick any data structure you want.\n",
    "              If you want to keep it simple, you can store a list of tuples of (s, a, r, s') in self._storage\n",
    "              However you may find out there are faster and/or more memory-efficient ways to do so.\n",
    "        \"\"\"\n",
    "        self._storage = deque([])\n",
    "        self._maxsize = size\n",
    "        \n",
    "        # OPTIONAL: YOUR CODE\n",
    "        \n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self._storage)\n",
    "\n",
    "    def add(self, obs_t, action, reward, obs_tp1, done):\n",
    "        '''\n",
    "        Make sure, _storage will not exceed _maxsize. \n",
    "        Make sure, FIFO rule is being followed: the oldest examples has to be removed earlier\n",
    "        '''\n",
    "        data = (obs_t, action, reward, obs_tp1, done)\n",
    "        \n",
    "        # add data to storage\n",
    "        self._storage.append(data)\n",
    "        if self.__len__() > self._maxsize:\n",
    "            self._storage.popleft()\n",
    "        \n",
    "    def sample(self, batch_size):\n",
    "        \"\"\"Sample a batch of experiences.\n",
    "        Parameters\n",
    "        ----------\n",
    "        batch_size: int\n",
    "            How many transitions to sample.\n",
    "        Returns\n",
    "        -------\n",
    "        obs_batch: np.array\n",
    "            batch of observations\n",
    "        act_batch: np.array\n",
    "            batch of actions executed given obs_batch\n",
    "        rew_batch: np.array\n",
    "            rewards received as results of executing act_batch\n",
    "        next_obs_batch: np.array\n",
    "            next set of observations seen after executing act_batch\n",
    "        done_mask: np.array\n",
    "            done_mask[i] = 1 if executing act_batch[i] resulted in\n",
    "            the end of an episode and 0 otherwise.\n",
    "        \"\"\"\n",
    "#         idxes = <randomly generate batch_size integers to be used as indexes of samples>\n",
    "        idxes = np.random.randint(low=0, high=self.__len__(), size=batch_size)\n",
    "        \n",
    "        # collect <s,a,r,s',done> for each index\n",
    "        states = [ self._storage[idx][0] for idx in idxes ]\n",
    "        actions = [ self._storage[idx][1] for idx in idxes ]\n",
    "        rewards = [ self._storage[idx][2] for idx in idxes ]\n",
    "        next_states = [ self._storage[idx][3] for idx in idxes ]\n",
    "        is_done = [ self._storage[idx][4] for idx in idxes ]\n",
    "        \n",
    "        return np.array(states), np.array(actions), np.array(rewards), np.array(next_states), np.array(is_done)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "from collections import deque\n",
    "\n",
    "from SumTree import SumTree\n",
    "from segment_tree import SumSegmentTree, MinSegmentTree\n",
    "\n",
    "class PrioritizedReplayBuffer(object):\n",
    "    def __init__(self, size):\n",
    "        \"\"\"\n",
    "        Create Replay buffer.\n",
    "        Parameters\n",
    "        ----------\n",
    "        size: int\n",
    "            Max number of transitions to store in the buffer. When the buffer\n",
    "            overflows the old memories are dropped.\n",
    "            \n",
    "        Note: for this assignment you can pick any data structure you want.\n",
    "              If you want to keep it simple, you can store a list of tuples of (s, a, r, s') in self._storage\n",
    "              However you may find out there are faster and/or more memory-efficient ways to do so.\n",
    "        \"\"\"\n",
    "        self._storage = []\n",
    "        self._maxsize = size\n",
    "        self._next_idx = 0\n",
    "        \n",
    "        it_capacity = 1\n",
    "        while it_capacity < size:\n",
    "            it_capacity *= 2\n",
    "\n",
    "        self._it_sum = SumSegmentTree(it_capacity)\n",
    "\n",
    "        \n",
    "        self._max_priority = 1.0\n",
    "        \n",
    "        self.epsilon = 1e-6 # small positive constant\n",
    "        self._alpha = 0.6\n",
    "        self.beta = 0.4\n",
    "        self.beta_increment_per_sampling = 0.001\n",
    "        \n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self._storage)\n",
    "    \n",
    "    def add(self, obs_t, action, reward, obs_tp1, done):\n",
    "        data = (obs_t, action, reward, obs_tp1, done)\n",
    "        idx = self._next_idx\n",
    "        \n",
    "        if self._next_idx >= len(self._storage):\n",
    "            self._storage.append(data)\n",
    "        else:\n",
    "            self._storage[self._next_idx] = data\n",
    "        self._next_idx = (self._next_idx + 1) % self._maxsize\n",
    "        \n",
    "        self._it_sum[idx] = self._max_priority ** self._alpha\n",
    "    \n",
    "    def _sample_proportional(self, batch_size):\n",
    "        res = []\n",
    "#         p_total = self._it_sum.sum(0, len(self._storage) - 1)\n",
    "        p_total = self._it_sum.sum(0, len(self._storage))\n",
    "#         print (p_total)\n",
    "        every_range_len = p_total / batch_size\n",
    "        for i in range(batch_size):\n",
    "            mass = random.random() * every_range_len + i * every_range_len\n",
    "            idx = self._it_sum.find_prefixsum_idx(mass)\n",
    "            res.append(idx)\n",
    "        return res\n",
    "    \n",
    "    def _encode_sample(self, idxes):\n",
    "        obses_t, actions, rewards, obses_tp1, dones = [], [], [], [], []\n",
    "        for i in idxes:\n",
    "            data = self._storage[i]\n",
    "            obs_t, action, reward, obs_tp1, done = data\n",
    "            obses_t.append(np.array(obs_t, copy=False))\n",
    "            actions.append(np.array(action, copy=False))\n",
    "            rewards.append(reward)\n",
    "            obses_tp1.append(np.array(obs_tp1, copy=False))\n",
    "            dones.append(done)\n",
    "        return np.array(obses_t), np.array(actions), np.array(rewards), np.array(obses_tp1), np.array(dones)\n",
    "        \n",
    "    def sample(self, batch_size):\n",
    "        \n",
    "        beta = np.min([1., self.beta + self.beta_increment_per_sampling]) # linear schedule\n",
    "        idxes = self._sample_proportional(batch_size)\n",
    "\n",
    "        weights = []\n",
    "#         p_min = self._it_min.min() / self._it_sum.sum() # gloal max weight\n",
    "#         max_weight = (p_min * len(self._storage)) ** (-beta)\n",
    "\n",
    "        for idx in idxes:\n",
    "            p_sample = self._it_sum[idx] / self._it_sum.sum()\n",
    "            weight = (p_sample * len(self._storage)) ** (-beta)\n",
    "#             weights.append(weight / max_weight)\n",
    "            weights.append(weight)\n",
    "\n",
    "        weights = np.array(weights)\n",
    "        weights = weights / weights.max()\n",
    "        encoded_sample = self._encode_sample(idxes)\n",
    "        return tuple(list(encoded_sample) + [weights, idxes])\n",
    "    \n",
    "    def update_priorities(self, idxes, priorities):\n",
    "        \"\"\"Update priorities of sampled transitions.\n",
    "        sets priority of transition at index idxes[i] in buffer\n",
    "        to priorities[i].\n",
    "        Parameters\n",
    "        ----------\n",
    "        idxes: [int]\n",
    "            List of idxes of sampled transitions\n",
    "        priorities: [float]\n",
    "            List of updated priorities corresponding to\n",
    "            transitions at the sampled idxes denoted by\n",
    "            variable `idxes`.\n",
    "        \"\"\"\n",
    "        assert len(idxes) == len(priorities)\n",
    "        for idx, priority in zip(idxes, priorities):\n",
    "            assert priority > 0\n",
    "            assert 0 <= idx < len(self._storage)\n",
    "            self._it_sum[idx] = priority ** self._alpha\n",
    "\n",
    "            self._max_priority = max(self._max_priority, priority)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some tests to make sure your buffer works right"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success!\n"
     ]
    }
   ],
   "source": [
    "replay = ReplayBuffer(2)\n",
    "obj1 = tuple(range(5))\n",
    "obj2 = tuple(range(5, 10))\n",
    "replay.add(*obj1)\n",
    "assert replay.sample(1)==obj1, \"If there's just one object in buffer, it must be retrieved by buf.sample(1)\"\n",
    "replay.add(*obj2)\n",
    "assert len(replay._storage)==2, \"Please make sure __len__ methods works as intended.\"\n",
    "replay.add(*obj2)\n",
    "assert len(replay._storage)==2, \"When buffer is at max capacity, replace objects instead of adding new ones.\"\n",
    "assert tuple(np.unique(a) for a in replay.sample(100))==obj2\n",
    "replay.add(*obj1)\n",
    "assert max(len(np.unique(a)) for a in replay.sample(100))==2\n",
    "replay.add(*obj1)\n",
    "assert tuple(np.unique(a) for a in replay.sample(100))==obj1\n",
    "print (\"Success!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success!\n"
     ]
    }
   ],
   "source": [
    "replay = PrioritizedReplayBuffer(2)\n",
    "\n",
    "obj1 = tuple(np.array(range(5)) * 1)\n",
    "obj2 = tuple(np.array(range(5)) * 2)\n",
    "\n",
    "replay.add(*obj1) # why only one element will occur error\n",
    "\n",
    "assert replay.sample(1)[:5] == obj1, \"If there's just one object in buffer, it must be retrieved by buf.sample(1)\"\n",
    "replay.add(*obj2)\n",
    "assert len(replay._storage) == 2, \"Please make sure __len__ methods works as intended.\"\n",
    "replay.add(*obj2)\n",
    "assert len(replay._storage) == 2, \"When buffer is at max capacity, replace objects instead of adding new ones.\"\n",
    "assert tuple(np.unique(a) for a in replay.sample(100))[:5] == obj2\n",
    "replay.add(*obj1)\n",
    "assert max(len(np.unique(a)) for a in replay.sample(100)) == 2\n",
    "replay.add(*obj1)\n",
    "assert tuple(np.unique(a) for a in replay.sample(100))[:5] == obj1\n",
    "print (\"Success!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's use this buffer to improve training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "from qlearning import QLearningAgent\n",
    "\n",
    "env = gym.make(\"Taxi-v2\")\n",
    "n_actions = env.action_space.n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def play_and_train_with_replay(env, agent, replay=None, \n",
    "                               t_max=10**4, replay_batch_size=32, prioritized=False):\n",
    "    \"\"\"\n",
    "    This function should \n",
    "    - run a full game, actions given by agent.getAction(s)\n",
    "    - train agent using agent.update(...) whenever possible\n",
    "    - return total reward\n",
    "    :param replay: ReplayBuffer where agent can store and sample (s,a,r,s',done) tuples.\n",
    "        If None, do not use experience replay\n",
    "    \"\"\"\n",
    "    total_reward = 0.0\n",
    "    s = env.reset()\n",
    "    \n",
    "    for t in range(t_max):\n",
    "        # get agent to pick action given state s\n",
    "        a = agent.get_action(s)\n",
    "        \n",
    "        next_s, r, done, _ = env.step(a)\n",
    "\n",
    "        # update agent on current transition. Use agent.update\n",
    "        agent.update(s, a, r, next_s)\n",
    "        \n",
    "        if replay is not None:\n",
    "            # store current <s,a,r,s'> transition in buffer\n",
    "            replay.add(s, a, r, next_s, done)\n",
    "            \n",
    "            # sample replay_batch_size random transitions from replay, \n",
    "            # then update agent on each of them in a loop\n",
    "            if prioritized:\n",
    "                states, actions, rewards, next_states, is_done, weights, idxes = replay.sample(replay_batch_size)\n",
    "            else:\n",
    "                states, actions, rewards, next_states, is_done = replay.sample(replay_batch_size)\n",
    "            td_errors = [] \n",
    "            for idx in range(replay_batch_size):\n",
    "                td_error = agent.update(states[idx], actions[idx], rewards[idx], next_states[idx])\n",
    "                td_errors.append(np.abs(td_error)+1e-6)\n",
    "#             print (idxes, td_errors)\n",
    "            if prioritized:\n",
    "                replay.update_priorities(idxes, td_errors)\n",
    "        s = next_s\n",
    "        total_reward +=r\n",
    "        if done:break\n",
    "    \n",
    "    return total_reward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create two agents: first will use experience replay, second will not.\n",
    "\n",
    "agent_baseline = QLearningAgent(alpha=0.5, epsilon=0.25, discount=0.99,\n",
    "                       get_legal_actions = lambda s: range(n_actions))\n",
    "\n",
    "agent_replay = QLearningAgent(alpha=0.5, epsilon=0.25, discount=0.99,\n",
    "                       get_legal_actions = lambda s: range(n_actions))\n",
    "\n",
    "replay = ReplayBuffer(1000)\n",
    "prioritized = False\n",
    "# replay = PrioritizedReplayBuffer(1000)\n",
    "# prioritized = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline : eps = 2.9191091959171894e-05 mean reward = 9.7\n",
      "ExpReplay: eps = 2.9191091959171894e-05 mean reward = 6.6\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD8CAYAAAB6paOMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VOW9+PHPdyYzWSHsYQkaULYAEWjYtNq4FNCrqNVWrQvVWlxvt1tb/XmvWlut1/betnpdSitqr7baoq2UasUt9YILi+LCHkmAAAIJJGSyTGZ5fn+ck2QCCVlmS3K+79drXjPznDPnfOfJZL7zPOec5xFjDEoppZzLlewAlFJKJZcmAqWUcjhNBEop5XCaCJRSyuE0ESillMNpIlBKKYfTRKCUUg6niUAppRxOE4FSSjlcSrID6IwhQ4aYvLy8br++traWzMzM2AXUi2ldtKb10ZrWR4u+UBfr16+vMMYM7Wi9XpEI8vLyWLduXbdfX1xcTFFRUewC6sW0LlrT+mhN66NFX6gLEdnZmfW0a0gppRxOE4FSSjmcJgKllHI4TQRKKeVwmgiUUsrhNBEopZTDaSJQSimHS9p1BCKyAPg14AZ+Z4x5IFmxKNUZxhj8wTBul+Bxu5rLjIGwMYTt+5bnVhlh+zHQNDGsAJ4UF163dXO5BIyBoB/jryHoryUYaCQYChEKhQmFgxAOkyIQNiGCwRChUAgxYYi4Wc8NQhjBICaMEELsMoxVjjG4mh+3rNN/byl1GyoRBFyCpGRgXG4CwRAhO/i01FSMO5WQeAhjkEA9rnAAlwniMmGCwQDBUBATCpDqhjSXAROy3p8IBP0QqLNidnshfSAmMwcCdUijD0KNVt0215T9zH6Ppvl92rdwuNUyTNh6SUqatX2XB8SFEUBc1jbFZd1c1r3B1VJm3/pVbKJxaxDjSSeYkgUYRAQxxtoEQDhoxWuM9VlojtWK14QakXAQEw5BKIiEAphwwCozYQwQDocJhw1eCeMmhJgQLhNAwiEkHIR+I3AVLorjJztJiUBE3MAjwJeBcmCtiCw3xmxKRjyq+4wx1AdCHKpt5HBtgPpACGMMIWPwB8I0BEI0BEPUN7Y89rpdZKam4BIQBBEQsf7hw2FDMGwIhcOEwtYXrz8YJhg2zcvCxhAMWfeBkLW81h/E5w9S6w+SlebBJRA2VnxhYwiHj/2SDoUNh6rr8a4vJhAyBEPWfggFSDN1DEppIDVcz5GQl5RgHZ5QPQPExyCpYaD4yMbHAHwMkFr6U0uG+EnHTxb1DJIaUmnEKyHrfRkhYP+7+fGQSoA9ZjCCIV0aycB6rUdCVpKwb4k2A2DbseXpiQ6kB/gCwKfW49QkxrEtZTzj+2IiAGYBJcaYHQAi8hxwIaCJIMkag2H2H2lgb1U9e6vr2VvVQKWvkcN1TbcA1XWN1DaGaAiE8AfCNIbCCYnN7RLcIta9S3AJeO1f1ZmpKWSmppDhdVNd14jLhBhqKhgV2svI8D5GhvaSburwu9IZHtrLwNAhMsM+POF6Ag3WMAJp4XrSwrV4jPWLlEDEzlM45r8ljJuGlH74Pf3xp/Qn4B5A0J1GY0omZZ4BhNzphMVt/YoE3CaAC4M77CckKWTU7SEgHnySRqOk4pc0gimZBFMyMSlpSIoXcblxudyIyw0iBMOCuFy43W7E5QJcGHEhYt2HEYz9C7e5XSAuwsaFkaZ2gr1MBGNchBDCdtmeshJOGD2q+RetO9QAJkxqirXPcDhEINBISjiAxwSsH/juNILiJYSLIG7c7hTcKR5cbje+RqgLGoy4sX5DG0KSSsCdBgguE6S/fx/eUB0BdzqN7gxCLi+A1aIB69d203sSAVzgst+HWHEbcdv31q95weAOB3CH/bhM0N6O3UpqbgWFm1tKzfeEcZkwEGbfnnJGjRyBJ1hHmmkAWlp0YcT64e9yExKP3dJo+UFjNRmEsMtLWNxWnC4PRlIIu1IwrhSM9TMIt9uF2+WiLgiNxkXQpBASF0HjIkgKOQMyGB+j/6H2JCsRjAJ2RzwvB2YnKRZHCRvD7kN17KioZcdBH59XN1Dha+Szgz72VtVz0OfHmNavyfS6GZjpZWCGlwEZHk4YlEGm102ax7oNyPAwyF6W7nXjEutXfrqnZZ00j6v5eUMgZLccWn6lN+3T7RZSXILL/sJPTXFZX0Iuifgns/lr4NAOOFwGh0rhcKl1X1sG1eVWd0QTTwZ4s6wuiYFjIPtkSBvA3oOHGTk4y/pH9mZCan9I7Wetm9bf6l4I1NtlmZA+CDIGQcZgXKn9yBAhI55/sAQrLg73+mEVYqUvDDHRWclKBNJGWauvHxFZDCwGyMnJobi4uNs78/l8Ub2+NwobQ2W9odwXZkdVmH21YQ7WG/b6QgRefat5vRQXZHmEEZnChP4u5g7zMDhdGJTmYlCaMDhNSE1p+nMF7Vt9650ZoNa6hbBuAI1AdQzeiyvUSEbdLrJ8ZWTWlpHl20l6/V7S/BWt1mv09KchbTj16SfSMHoW9ek51KePoD59BI3eQVb/9FF8Xh/bsrKOfT9++3aMGvvWqSFceh0n/q+0x0l1kaxEUA6MjnieC+yNXMEYswRYAlBYWGiiycx9PbPXN4b4/EgDH+2u4sNdh9mwu4rNn9fQGLS6bNwu4cRBGZw4PIOJjYcpmjGJsUMyGTs0iyFZ3mN/aSeL3we73oUje6GhGvZtgM8/hcrtEQcA02HYJDjhbBg8DoaOt37hD8zDm9YfL9C/C7vs65+NrtL6aOGkukhWIlgLjBORMcAe4HLg60mKpVfa+nkNKz7eyyd7qlldUkHAPqUjw+umIDebRXNP5KShWZw8LIvJI7NJ97oB+8M964Rkht4iUA/7PoZPl8Fnb0LlZ7RqGGaPhuEFkH8h5EyGnCkwaAy43EkLWam+KCmJwBgTFJFbgVexTh9daozZmIxYepMKn5+XNuzlxQ/K2bj3CCIwemAGV8/JY3xOFqeMHsC4YVmkuHvo5SF1h2DTX6GixOrb3/EWBBusfvixZ8LUr0FuIaQNgAEnQFaHw6grpWIgadcRGGNeBl5O1v57i4ZAiDe3HOCF9eUUbztIKGyYOiqbey7I54JTRjI4K5kntnVCYy3sfh8++F/Y8ncI+a0Dt/1GwPSrYPRsGL/AOjCrlEqKXjExjRPtP9LAo2+V8JcP93CkIUhO/1SuP30Ml8zIZXxOv2SH1z5jmi+wYd0T8H//BXWVkJYNX1gEM66xunh6ynEJpZQmgp4mFDYsXVXKL1/fRiAU5rypI7hkRi6nnTwEt6uHfnnWH4b9m6CuAt5fAjtXWQd1g/VWl8/sG2DMGdbpl0qpHkcTQQ/y6sbP+fmrWyk54OOcScP4j/PzOXFwD/7yPLIP3v45fPRH6/x8sLp9PJlw4qlw6r/C2C8lN0alVIc0EfQAB2oaeODlLbz44R4m5PTjf74+nX+ZOqLnnNYZyRgo/Sd8sgw+fQFCAZjyFTj5y1Y//5gvgSct2VEqpbpAE0GSvfzJPm7780f4g2G+fdbJ3HrWOLwpPfSsn8ZaeOlW2PgiuFIg/yI469+tUzqVUr2WJoIkCYTC/PhvG3nmvV1MGz2AX102jbwhPawbyBjYv9EawXHPOnjnYTiwGWbfCKd9B/qPTHaESqkY0ESQBKUVtXz3uQ/5qLyab50+htvmT+x5rYDDZbDi+/DZGy1laQPgymUw7pykhaWUij1NBAn2cXkVV/3ufUSE//7aKXxlRm6yQzrW1n/AC98ExLqwq2oXnP9LKLgcvH1piDWlFGgiSKjt+2u4ZukasjM8PPvNOZwwuAd+qX6yDP5yg3Wu/1ef0v5/pRxAE0GCHKpt5Lqn1+Jxu3jmm7N7ZhJ486fW6aAnfhGu+KNe7auUQ2giSIBAKMwtz37A/iN+/nTD3J53bcCRfbD6V/D+41BwGVzwa/A4cU4qpZxJE0EC/HTFJt7dUcl/ffUUpo0ekOxwWqv8DJbOh9qD1umgFz1uzeOqlHIMTQRxtuTtz3j63Z1c/8UxXPKFHnZguO4QPPtVCDTABQ/BtCs1CSjlQJoI4ujTPdU8+I+tLJg8nNvPnZjscFrb+S68dDNU74FFy+GEOcmOSCmVJPrzL04ag2F+8OePGJjp5YFLpvasOQJ2/BOe+hcINsKVf9YkoJTDaYsgTh5+cztbPq/hd9cUMiDDm+xwmg2qXAfv/BIGnwzXvgyZQ5IdklIqyTQRxMFHu6t4tPgzLpmRyzn5OckOx2IMrPguBZ88BekD4bJnNAkopQDtGoq5QCjMbcs+YkiWl7suyE92OC02vQTrn2L/sDPge5usSd+VUgpNBDH3xzW72Lbfx08unEJ2uifZ4VjqD8MrP4QRp7Bl4nd1mAilVCuaCGKopiHAr1/fzuwxg/hysruEjIGyVdbgca/dDbUVcMFDGJc7uXEppXocPUYQQ/f+bROH6hq5818mJX9SmTd+DKt+2fL81H+FkdNgW3HSQlJK9UyaCGJk094jLPugnMWnj6UgN4lXDwfq4Z8Ptk4CAEV3JCcepVSPp4kgBsJhw10vfUq/1BRuLjo5OUEEGuDFb8H2lRBssMpufh+WXQsLHtCJ45VS7dJEEANvbjnAup2HeeArU8nOSNIB4vcfg83LW55/9WkYNhFufjc58Sileg1NBDHw+/d2Mrx/GpcmayyhukPwfxFdQT8shYxByYlFKdXrRHXWkIh8VUQ2ikhYRAqPWnaHiJSIyFYRmR9RvsAuKxGR26PZf0+wq7KOt7cd5PJZo5MzjIQxsOJ7EKiFm96BO/drElBKdUm031yfAl8B3o4sFJF84HJgMrAAeFRE3CLiBh4BzgXygSvsdXutZ97fidslXD7zhOQEsPLfYdNfoeh2yJkMnrTkxKGU6rWi6hoyxmwG2jpV8kLgOWOMHygVkRJglr2sxBizw37dc/a6m6KJI1nqGoM8t2YXC6YMZ3h2Er6A3/kfePd/YPJX4IvfT/z+lVJ9Qrz6MkYBuyOel9tl7ZX3Sss37OVIQ5BvnJqX+J1/sgxW3mlNJnPxb0AvFFNKdVOHLQIReR0Y3saiO40xL7X3sjbKDG0nHtPOfhcDiwFycnIoLi7uKNR2+Xy+qF7fnqXv1zMyU/CVfkRxWWIuIBtw+BMya3dx4s7naeg3jg+HXINZ9U6nXx+vuuittD5a0/po4aS66DARGGPO6cZ2y4HREc9zgb324/bKj97vEmAJQGFhoSkqKupGGJbi4mKieX1bdlbWsu0fxfxwwQTOTNS1Awe3wSMX2k8E7zdf5kvDp3RpE/Goi95M66M1rY8WTqqLeHUNLQcuF5FUERkDjAPWAGuBcSIyRkS8WAeUlx9nOz3Wix/sQQQunp6gni1j4JXbrMfTroSvPgldTAJKKdWWqA4Wi8jFwMPAUODvIrLBGDPfGLNRRP6EdRA4CNxijAnZr7kVeBVwA0uNMRujegdJ8reP9zJ37GBGZKfHf2fhEKz4LuwohjP/Hb50W/z3qZRyjGjPGvoL8Jd2lt0H3NdG+cvAy9HsN9lKK2rZcbCWRXPz4r+zhmr4zRnWKKKTL4bTvhP/fSqlHEWvLO6GNzbvB+DsScPiu6OyVbDxr1YSGFsEX30qvvtTSjmSJoJueH3zfiYO70fuwDhO8PLXm2HDs9bj8efC15+L376UUo6mE9N0UXV9gLVlh+PbGtj5bksSADhdLxZTSsWPtgi66N3PKgmFDV8aH6dEEA7Dy7dZE8zfuAr6jQSX5mulVPxoIuii1SUVZHrdTD8hTpPPbH4J9n8CX/ktZCdpNFOllKPoT80uWl1Swawxg/DEY6TRxjp47S4YOhGmXBL77SulVBs0EXTB3qp6dlTUctrJQ2K/cWPgn/8JVbvg3P/UsYOUUgmjiaALVpdUAPDFcXFIBKVvw+pfwcTzYcyXYr99pZRqhyaCLlhdUsGQLC8TcvrFfuNrf2cdIL7kd3DssN5KKRU3mgg6yRjDO59VMvekIW3NvxCdD5+15huesQg8CRiyQimlImgi6KTSiloO1PiZO3ZwbDe8ew28dDN4+8Gp347ttpVSqhM0EXTSezsOATBnbIznA37vURAX/Os6yIxxklFKqU7QRNBJH+w6zKBML2OGZMZuo/WHYfMKmH0T9Gtr7h+llIo/TQSd9HF5FafkZsf2+MDqhyAchIKvxm6bSinVRXplcSf4/EG2H/Bx7pQRsdngkb3wxDyo3g0nnQ0jp8dmu0op1Q2aCDrh0z3VGAPTRsdgWImGavjvSS3Pz7kn+m0qpVQUNBF0wke7qwAoyM2OfmMb7Xl8xG0NKpeTH/02lVIqCpoIOuHj8mpyB6YzOCs1+o1tfw0yBsMPtuswEkqpHkEPFnfAGMMHuw5zSiy6hRpr4bO3IP8iTQJKqR5DE0EHdh2qY191Q2wuJNv6CgRqYcpXot+WUkrFiCaCDmzaewSAU3Jj0CLY+gpkDoMTTo1+W0opFSOaCI6jqq6RrftrEIGTh2VFtzFjYNe7kHeazjimlOpR9GBxOzbsruKiR1bjEjhxUAbp3ij79Kt2wZE92hpQSvU4+tO0Het3HgYgbGBcLIad/vh56/6ks6LfllJKxZAmgnZU1weaH08cHmUiCDTAmiUwbh4MOTnKyJRSKraiSgQi8nMR2SIiH4vIX0RkQMSyO0SkRES2isj8iPIFdlmJiNwezf7jqbSitvnx+GhbBDvegtqDMPuGKKNSSqnYi7ZF8BowxRhTAGwD7gAQkXzgcmAysAB4VETcIuIGHgHOBfKBK+x1e5zSCl/z4wnRtghK3wZXCpx4WpRRKaVU7EWVCIwxK40xQfvpe0Cu/fhC4DljjN8YUwqUALPsW4kxZocxphF4zl63RzHGUHqwpUUQ1dDToSB88HsYv0BnH1NK9UixPGvoOsA+IsoorMTQpNwuA9h9VPnsGMYQEwdr/NQ2hrj93ImcO2U4HncU+fLARmj0weSLYxegUkrFUIeJQEReB9qaNeVOY8xL9jp3AkHg2aaXtbG+oe0WiGlnv4uBxQA5OTkUFxd3FGq7fD5fl16/5VAIgODBMko/2U1pt/cMI/e8wnjgvfIQDZWdjyFeuloXfZ3WR2taHy2cVBcdJgJjzDnHWy4ii4DzgbONMU1f6uXA6IjVcoG99uP2yo/e7xJgCUBhYaEpKirqKNR2FRcX05XX731/F/AJF509l9yBGd3eLwB/fgqycpiz4GsQ60nvu6GrddHXaX20pvXRwkl1Ee1ZQwuAHwELjTF1EYuWA5eLSKqIjAHGAWuAtcA4ERkjIl6sA8rLo4khHkorfHhTXIzMjrJP3++Drf+Aief3iCSglFJtifYYwf8AqcBr9hSO7xljbjTGbBSRPwGbsLqMbjHGhABE5FbgVcANLDXGbIwyhpgrragjb3AGLleUX94b/gDBepiqU1EqpXquqBKBMabdq6OMMfcB97VR/jLwcjT7jacDNQ1s3neEySP7R7+xTS/B8Klwwpzot6WUUnGiVxYfZdZ9b7Cnqj66U0bBOm107wfWtQPaLaSU6sE0EUTwB0PNj/OiTQQHNkKgDnJnRhmVUkrFlyaCCAeO+Jsf5w2OMhHsXmPdayJQSvVwmggi7KtuaH4cVdeQMbD+KcgaDgNOiD4wpZSKI00EEfZV1zc/zukfxUT1HzwN+z+F2Yv1+IBSqsfTRBBh/xGrRfD0dbOQ7n6BN9bC375jPZ7U44ZRUkqpY2giiLCvuoFMr5svjR/avQ1U74H7R1qPZ9+kcw8opXoFTQQRDhzxk9M/rfsbWPG9lsfzfhp9QEoplQCaCCIcqGlgaL9uHhsIh2DnOzCqEG5+H9w6HbRSqnfQRBDhYI2fYd1tEaz+NTTWwOwbYdjE2AamlFJxpIkgwoEaP8O62yJY96R1P+aM2AWklFIJoInA5vMHqWsMda9ryBhoqIaZ10O/nNgHp5RScaSJwHbAPnW0Wy2Cf9wB/moYcUqMo1JKqfjTRGA7WGMNLzGsXxePEdRWwvuPgbggX68bUEr1PpoIbAeaEkFXryjebU/NfO0rkJYd46iUUir+NBHYmhNBV7uG9n1ktQa0W0gp1UtpIrAdqGnA63aRne7p2gv3b4RBJ4EnymktlVIqSTQR2A7W+BnaL7XrYwwd2AQ5+fEJSimlEkATga3S18iQLG/XXrR5BRzaAUMmxCcopZRKAE0Etgqfn8FZXTw+sPMd677w2tgHpJRSCaKJwFbh83e9RXBwCwwvgP4j4xOUUkolgCYCwBhDpa+x6y2Cim0wVMcVUkr1bpoIgOr6AMGwYUhXEoG/Bqp3w1A9PqCU6t00EWB1CwFd6xqq2Gbda4tAKdXLaSIAKnyNAF1rEez5wLrXRKCU6uWiSgQi8hMR+VhENojIShEZaZeLiDwkIiX28hkRr1kkItvt26Jo30AstLQIOpkIfAfg5R9AzhQYNDaOkSmlVPxF2yL4uTGmwBgzDVgB3GWXnwuMs2+LgccARGQQcDcwG5gF3C0iA6OMIWqVdotgcGe7htYtte5P/z64tFGllOrdovoWM8YciXiaCRj78YXA743lPWCAiIwA5gOvGWMOGWMOA68BC6KJIRYqfH5cAgMzOpkI9n1stQSmXBLfwJRSKgGinlhXRO4DrgGqgTPt4lHA7ojVyu2y9srb2u5irNYEOTk5FBcXdztGn8933Nd/tM1Plgf+7+1/dritlICPL279O5/nnMmWKGJKlo7qwmm0PlrT+mjhpLroMBGIyOvA8DYW3WmMeckYcydwp4jcAdyK1fXT1oA95jjlxxYaswRYAlBYWGiKioo6CrVdxcXFHO/1S7a/x9icEEVFp3W8sT9/A4DhRd9k+KTux5QsHdWF02h9tKb10cJJddFhIjDGnNPJbf0B+DtWIigHRkcsywX22uVFR5UXd3L7cVNaUcvcsYM7XjEUhO2vwbQrYdIF8Q9MKaUSINqzhsZFPF0IbLEfLweusc8emgNUG2P2Aa8C80RkoH2QeJ5dljTGGA7U+Bme3YmZySq3Q6MPxnwp/oEppVSCRHuM4AERmQCEgZ3AjXb5y8B5QAlQB1wLYIw5JCI/Adba691rjDkUZQxRqW0MEQqbzs1DcGCTda/DTiul+pCoEoExps3TZowxBrilnWVLgaXR7DeWjtQHAOjfqUSwBcQNg8d1vK5SSvUSjj8J/kiDlQg63SIYNBY8XZzgXimlejDHJ4LqOrtFkNaJRHBwqw4yp5TqcxyfCI40BAHon95BL1koAIdLYcj4BESllFKJo4mgvpNdQ4d3QjgIQ/T4gFKqb9FE0NDJrqHK7da9HihWSvUxmgjqra6hfmkddA0dtC+RGHxSnCNSSqnEcnwiqK4PkOl1k+LuoCq2vwbDJkPGoMQEppRSCeL4RHCkIdC5U0crtkHuF+IfkFJKJZgmgvpAxxeTBf1QexCyRx9/PaWU6oU0ETQEOj5QXGWPnN2/zRGzlVKqV9NEUB/suEXw2RvWfe7M+AeklFIJ5vhEUF0f6Phisg1/gOEFeg2BUqpPcnQiOFDTwJ6q+uN3DRljDS2RdzpIW/PqKKVU7+boRLCu7DAA43Ky2l/Jtx+C9TBoTIKiUkqpxHJ0IvD5rYvJzhg3tP2V9nxg3etgc0qpPsrRiaDOTgSZqcc5RvDZG+DJgNGzExSVUkollqMTQW1jCIAMr7vtFYyBktet4wMpqQmMTCmlEsfZicAfxO0SUlPaqYYDm+FwGUxYkNC4lFIqkRydCOoaQ2R63Uh7ZwNtf9W6H39u4oJSSqkEc3QiqPUHj3984NAOyMqB/iMSF5RSSiWYsxNBY7D94wMANZ9Dv+GJC0gppZLA2YnAHyLreC2Cmn3QT1sDSqm+zdGJoK4xSIa3nUQQDkHlDhiYl9CYlFIq0RydCGr9ITJT2+kaOrQDArXWGENKKdWHxSQRiMgPRMSIyBD7uYjIQyJSIiIfi8iMiHUXich2+7YoFvvvrtrjtQj2fWTdj9BEoJTq2zoYdrNjIjIa+DKwK6L4XGCcfZsNPAbMFpFBwN1AIWCA9SKy3BhzONo4usNqERwnEbi9MHRiYoNSSqkEi0WL4JfAD7G+2JtcCPzeWN4DBojICGA+8Jox5pD95f8akLSrteoag2S2d9bQ5x/DsEng7sQ0lkop1YtFlQhEZCGwxxjz0VGLRgG7I56X22XtlSdcMBSmrjFEVlobLQJjrBbBiFMSH5hSSiVYh11DIvI60NbJ9HcC/w+Y19bL2igzxylva7+LgcUAOTk5FBcXdxRqu3w+3zGvr2m0dnuwfCfFxXtbLUur/5w59YfZ6stkXxT77Ynaqgsn0/poTeujhZPqosNEYIw5p61yEZkKjAE+sodoyAU+EJFZWL/0I2d6zwX22uVFR5UXt7PfJcASgMLCQlNUVNTWap1SXFzM0a8vraiFN4spPGUSRdNzrcJd78PGv1jXDwATii5jwsjp3d5vT9RWXTiZ1kdrWh8tnFQX3T5YbIz5BBjW9FxEyoBCY0yFiCwHbhWR57AOFlcbY/aJyKvA/SIy0H7ZPOCObkcfhaq6RgCyI+crXnpU42bY5ARGpJRSyRH1WUPteBk4DygB6oBrAYwxh0TkJ8Bae717jTGH4hTDcVXXBwDITvdaBQe3HbtSijeBESmlVHLELBEYY/IiHhvglnbWWwosjdV+u6tpdrJ+TQeLH5nZsjA1G065PAlRKaVU4sWrRdDjNQTCAKR7jjp9dP7PYM5NOlG9UsoxHDvERH3Amp0szeOGqogzWkdO0ySglHIUxyaCBnuaynSvG/ZvtAq//ic48dQkRqWUUonn3ETQ1CJIcUG9PcLFkHFJjEgppZLDsYmgPhDC4xZS3C5oqLIK0wYkNyillEoCRyeCtKYDxU0tgrTs5AWklFJJ4thE0BAItZwxVF9lJQHXcaatVEqpPsrBiSDcukWg3UJKKYdybCKob4xoETRUQfrA479AKaX6KOcmgkCING9E11C6tgiUUs7k2ETQEAhZp46Cdg0ppRzN0Ykg3atdQ0op5dhEUN901lCgAeoqISsn2SEppVRSODIRrNz4Odv2+6yzhiq3gwnD0AnJDksppZLCcaOPBkJhFv/vesCavJ7D5daCQWOTGJVSSiUqbQWwAAAR40lEQVSP41oETaOOApQc8FndQgAZg5MUkVJKJZfjEkFDRCKo9Yeg3p4gTROBUsqhnJcIGq0JaUYNSOd3iwqtFkFKGngzkhyZUkolh/MSQdBqEdxx3kSmjMqGLX+HzGFJjkoppZLHcYmgvrFpHgI31B2CQzsgf2GSo1JKqeRxXCJoOkaQ7nVD5WdWYd4XkxiRUkoll+MSQctcxS6otucqHnBiEiNSSqnkclwiaAhYB4vTPO6WU0czhyQxIqWUSi4HJoKmFoF9jAB0nCGllKM5LhH4/EEAslx+OPQZpGaD25PkqJRSKnmiSgQico+I7BGRDfbtvIhld4hIiYhsFZH5EeUL7LISEbk9mv13R62dCIb88Tz4+HnI1AvJlFLOFouxhn5pjPlFZIGI5AOXA5OBkcDrIjLeXvwI8GWgHFgrIsuNMZtiEEen1DaGAIO7YotV0H9UonatlFI9UrwGnbsQeM4Y4wdKRaQEmGUvKzHG7AAQkefsdROXCPxBLvGuaSkYmJeoXSvlKIFAgPLychoaGpIdSrdkZ2ezefPmZIfRKWlpaeTm5uLxdK+bOxaJ4FYRuQZYB/ybMeYwMAp4L2KdcrsMYPdR5bNjEEP7Xrubk3aWQlERYCWC8SmfQ9heXnRHXHevlFOVl5fTr18/8vLyEJFkh9NlNTU19OvXL9lhdMgYQ2VlJeXl5YwZM6Zb2+gwEYjI68DwNhbdCTwG/AQw9v1/AdcBbf3VDW0fkzDt7HcxsBggJyeH4uLijkJtU9HqXzEaml9fsquBwcYaV2jDKT+h6sPtwPZubbs38vl83a7Lvkjro7VY1kd2djaDBw/G5/PFZHuJFgqFqKmpSXYYneL1eqmqqur2367DRGCMOaczGxKR3wIr7KflwOiIxbnAXvtxe+VH73cJsASgsLDQFNm/6Lus2LorKiqiuj7A5jff4PKhXqiEaeddC6k9P+PHUnFxMd2uyz5I66O1WNbH5s2b6d+/f0y2lQy9pUXQJC0tjenTp3frtdGeNTQi4unFwKf24+XA5SKSKiJjgHHAGmAtME5ExoiIF+uA8vJoYuiKt7cdpLYxxKmj060Cj444qpRKrry8PCoqKpIaQ7THCB4UkWlY3TtlwA0AxpiNIvInrIPAQeAWY0wIQERuBV4F3MBSY8zGKGPotKq6RgAGeILgTgWXO1G7Vkr1ccFgkJSU3jnpY1QtAmPM1caYqcaYAmPMQmPMvohl9xljTjLGTDDGvBJR/rIxZry97L5o9t9VNfY1BN6wHzzpidy1UioJnnnmGWbNmsW0adO44YYbCIVC7Ny5k3HjxlFRUUE4HOb0009n5cqVlJWVMXHiRBYtWkRBQQFXX301dXV1x93+Pffcw+LFi5k3bx7XXHMNoVCI2267jZkzZ1JQUMBvfvMbwOpyO+OMM7j44ovJz8/nxhtvJBwOH7O9iy66iC984QtMnjyZJUuWAPDEE0/wve99r3md3/72t3z/+9+PYS05bM7iWn8Ql4A7VAfezGSHo5Rj/PhvG9m090hMt5k/sj93XzC53eWbN2/m+eefZ/Xq1Xg8Hm6++WaeffZZrrnmGn70ox9x4403Mnv2bPLz85k3bx5lZWVs3bqVJ554gtNOO42rr76aRx99lB/84AfHjWP9+vWsWrWK9PR0lixZQnZ2NmvXrsXv93Paaacxb948ANasWcOmTZs48cQTWbBgAS+++CKXXnppq20tXbqUQYMGUV9fz8yZM7nkkku4/PLLKSgo4MEHH8Tj8fDkk082J5hYcc4QE8ZQ6w/RL9WFbHgWjuxJdkRKqTh64403WL9+PTNnzmTatGm88cYb7NixA4Drr7+empoaHn/8cX7xi5brYUePHs1pp50GwGWXXcaqVas63M/ChQtJT7d6GFauXMnvf/97pk2bxuzZs6msrGT7duusxFmzZjF27FjcbjdXXHFFm9t+6KGHOOWUU5gzZw67d+9m+/btZGZmctZZZ7FixQq2bNlCIBBg6tSpUddPJMe0CA5VH+Hpd3bwZ++9Vvob2L3zbZVSXXe8X+7xYoxh0aJF/OxnPztmWV1dHeXl5YB1ymzT2UFHX+/QmesfMjNbeheMMTz88MPMnz+/1TrFxcUdbru4uJjXX3+dd999l4yMDIqKipovxrv++uu5//77mThxItdee22HMXWVY1oE2/57Phe5VlPo2mYVfPne5AaklIqrs88+m2XLlnHgwAEADh06xM6dOwH40Y9+xJVXXsm9997Lt771rebX7Nq1i3fffReAZcuW8cUvdm3Sqvnz5/PYY48RCAQA2LZtG7W1tYDVNVRaWko4HOb5558/ZtvV1dUMHDiQjIwMtmzZwnvvtVyTO3v2bHbv3s0f/vAHrrjiii7WRMcc0yKY49rMHG/E5eI6Wb1SfVp+fj4//elPmTdvHuFwGI/HwyOPPEJZWRlr165l9erVuN1uXnjhBZ588knOPPNMJk2axNNPP80NN9zAmDFjuOmmmwC46667KCwsZOHC409re/3111NWVsaMGTMwxjB06FD++te/AjB37lxuv/12Pvnkk+YDx5EWLFjA448/TkFBARMmTGDOnDmtln/ta19jw4YNDBwY+2HzHZMIjjGqMNkRKKXi7LLLLuOyyy47pjzy1/aLL74IQFlZGS6Xi8cffxywLijLyLB+MN57b9s9CPfcc0+r5y6Xi/vvv5/777//mHUzMjJ4/vnnjykvKytrfvzKK68cs7zJqlWrWp09FEuO6Rpq0pA6BO7cD+kDkh2KUkp1qKqqivHjx5Oens7ZZ58dl304qkVQbobw8blvcp4nLdmhKKV6mLy8PD799NOOV+yGoqKibg/dMWDAALZt2xbbgI7iqBbBzwJfx+PVJKCUUpEclQgCuPGmOOotK6VUhxz1rdhICl63o96yUkp1yFHfigFStEWglFJHcdS3YsCkkKqJQCnHKCsrY8qUKXHZdnFxMeeffz4Ay5cv54EHHojLfhLBUWcNaYtAKRUPCxcu7PBis56sT38r1vkDrZ7rMQKlnCcYDDYPLX3ppZdSV1fHvffey8yZM5kyZQqLFy/GGGvG3Iceeoj8/HwKCgr4xje+AUBtbS3XXXcdM2fOZPr06bz00kvH7OOpp57i1ltvBeAb3/gG3/72tzn11FMZO3Ysy5Yta17v5z//efMQ1XfffXf833wn9ekWQb0/QORAEgFSGDVQ5yFQKuFeuR0+/yS22xw+Fc7tuDsmcmjp6667jkcffZRbb72Vu+66C4Crr76aFStWcMEFF/DAAw9QWlpKamoqu3fvBuC+++7jrLPOYunSpVRVVTFr1izOOef4M/ju27ePVatWsWXLFhYuXMill17KypUr2b59O2vWrMEYw8KFC3n77bc544wzoq+LKPXpn8eDM1rPQCZuDx5tESjlKJFDS1911VWsWrWKt956i9mzZzN16lTefPNNNm60JkosKCjgyiuv5JlnnmmebWzlypU88MADTJs2rXlE0F27dh13nxdddBEul4v8/Hz279/fvJ2VK1cyffp0ZsyYwZYtW5qHqE62Pt0iIBxq9fTrp56cpECUcrhO/HKPl7aGf7755ptZt24do0eP5p577mke7vnvf/87b7/9NsuXL+fHP/4xmzdvxhjDCy+8wIQJE1ptp+kLvi2pqanNj5u6nYwx3HHHHdxwww2xemsx07d/HpuWRGDSB3LN2V9IYjBKqWSIHFr6j3/8Y/Pwz0OGDMHn8zX34YfDYXbv3s2ZZ57Jgw8+SHV1NT6fj/nz5/Pwww83f6F/+OGH3Ypj/vz5LF26FJ/PB8CePXuah8hOtj7eIrDmKC456ZucfNV/QScmmVBK9S2RQ0uPGzeOm266icOHDzN16lTy8vKYOXMmAKFQiKuuuorq6mqMMdxyyy0MGDCA//iP/+C73/0uBQUFGGPIy8tjxYoVXY5j3rx5bN68mblz5wKQlZXFM888w7Bhw2L6frtDmrJcT1ZYWGjWrVvX9RfWVsLPx7L95G8x7qpfdLy+AxQXF3d78Ku+SOujtVjWx+bNm5k0aVJMtpUMNTU1zTOX9QZt1beIrDfGdDjmft9uEbhTIP8i6t0jkx2JUkr1WH37GEFaNnztaQ4NnpHsSJRSqsfq24lAKaVUhzQRKKXipjccg+wLoq3nqBOBiPyriGwVkY0i8mBE+R0iUmIvmx9RvsAuKxGR26Pdv1KqZ0pLS6OyslKTQZwZY6isrCQtrfuTbkV1sFhEzgQuBAqMMX4RGWaX5wOXA5OBkcDrIjLeftkjwJeBcmCtiCw3xmyKJg6lVM+Tm5tLeXk5Bw8eTHYo3dLQ0BDVl2sipaWlkZub2+3XR3vW0E3AA8YYP4AxpunqiAuB5+zyUhEpAWbZy0qMMTsAROQ5e11NBEr1MR6PhzFjxiQ7jG4rLi5m+vTpyQ4jIaJNBOOB00XkPqAB+IExZi0wCngvYr1yuwxg91Hls9vasIgsBhYD5OTkUFxc3O0gfT5fVK/vS7QuWtP6aE3ro4WT6qLDRCAirwPD21h0p/36gcAcYCbwJxEZC7R1Ca+h7WMSbXYgGmOWAEvAuqAsmotc9KKhFloXrWl9tKb10cJJddFhIjDGtDveqojcBLxorKNBa0QkDAzB+qU/OmLVXGCv/bi9cqWUUkkQ1RATInIjMNIYc5d9MPgN4AQgH/gD1nGBkXb5OKyWwjbgbGAPsBb4ujFmYwf7OQjs7HagVnKqiOL1fYnWRWtaH61pfbToC3VxojFmaEcrRXuMYCmwVEQ+BRqBRXbrYKOI/AnrIHAQuMUYayhQEbkVeBVwA0s7SgIAnXkjxyMi6zoz3oYTaF20pvXRmtZHCyfVRVSJwBjTCFzVzrL7gPvaKH8ZeDma/SqllIodvbJYKaUczimJYEmyA+hBtC5a0/poTeujhWPqolfMR6CUUip+nNIiUEop1Y4+nQicOMCdiIwWkbdEZLM9EOB37PJBIvKaiGy37wfa5SIiD9l19LGI9LnJG0TELSIfisgK+/kYEXnfrovnRcRrl6faz0vs5XnJjDseRGSAiCwTkS32Z2SuUz8bIvI9+3/kUxH5o4ikOfWz0WcTgYi4sQa4OxfruoYr7MHw+rog8G/GmElYV3zfYr/v24E3jDHjsK7raEqM52Jd4zEOa0iPxxIfctx9B9gc8fw/gV/adXEY+KZd/k3gsDHmZOCX9np9za+BfxhjJgKnYNWL4z4bIjIK+DZQaIyZgnU6++U49bNhjOmTN2Au8GrE8zuAO5IdVxLq4SWs0V63AiPsshHAVvvxb4ArItZvXq8v3LCuXn8DOAtYgXVRYwWQcvTnBOv6lrn24xR7PUn2e4hhXfQHSo9+T078bGCNfbYbGGT/rVcA85362eizLQJa/tBNIge+cwS7+TodeB/IMcbsA7Dvh9mr9fV6+hXwQyBsPx8MVBljgvbzyPfbXBf28mp7/b5iLHAQeNLuKvudiGTiwM+GMWYP8AtgF7AP62+9Hod+NvpyImhv4DtHEJEs4AXgu8aYI8dbtY2yPlFPInI+cMAYsz6yuI1VTSeW9QUpwAzgMWPMdKCWlm6gtvTZ+rCPg1wIjMEaBicTqyvsaI74bPTlRHC8ge/6NBHxYCWBZ40xL9rF+0VkhL18BNA0d0RfrqfTgIUiUgY8h9U99CtggIg0XVUf+X6b68Jeng0cSmTAcVYOlBtj3refL8NKDE78bJwDlBpjDhpjAsCLwKk49LPRlxPBWmCcfRaAF+tA0PIkxxR3IiLAE8BmY8x/RyxaDiyyHy/COnbQVH6NfYbIHKC6qZugtzPG3GGMyTXG5GH9/d80xlwJvAVcaq92dF001dGl9vp95lefMeZzYLeITLCLzsYaD8xxnw2sLqE5IpJh/8801YUjPxtJP0gRzxtwHtZop58BdyY7ngS95y9iNVk/BjbYt/Ow+jPfALbb94Ps9QXr7KrPgE+wzqJI+vuIQ70UASvsx2OBNUAJ8Gcg1S5Ps5+X2MvHJjvuONTDNGCd/fn4K9Z8Io78bAA/BrYAnwL/C6Q69bOhVxYrpZTD9eWuIaWUUp2giUAppRxOE4FSSjmcJgKllHI4TQRKKeVwmgiUUsrhNBEopZTDaSJQSimH+//XfGV00bOwXAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import clear_output\n",
    "from pandas import DataFrame\n",
    "moving_average = lambda x, span=100: DataFrame({'x':np.asarray(x)}).x.ewm(span=span).mean().values\n",
    "\n",
    "rewards_replay, rewards_baseline = [], []\n",
    "\n",
    "for i in range(1000):\n",
    "    rewards_replay.append(play_and_train_with_replay(env, agent_replay, replay, prioritized=prioritized))\n",
    "    rewards_baseline.append(play_and_train_with_replay(env, agent_baseline, replay=None, prioritized=prioritized))\n",
    "    \n",
    "    agent_replay.epsilon *= 0.99\n",
    "    agent_baseline.epsilon *= 0.99\n",
    "    \n",
    "    if i %100 ==0:\n",
    "        clear_output(True)\n",
    "        print('Baseline : eps =', agent_replay.epsilon, 'mean reward =', np.mean(rewards_baseline[-10:]))\n",
    "        print('ExpReplay: eps =', agent_baseline.epsilon, 'mean reward =', np.mean(rewards_replay[-10:]))\n",
    "        plt.plot(moving_average(rewards_replay), label='exp. replay')\n",
    "        plt.plot(moving_average(rewards_baseline), label='baseline')\n",
    "        plt.grid()\n",
    "        plt.legend()\n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create two agents: first will use experience replay, second will not.\n",
    "\n",
    "agent_baseline = QLearningAgent(alpha=0.5, epsilon=0.25, discount=0.99,\n",
    "                       get_legal_actions = lambda s: range(n_actions))\n",
    "\n",
    "agent_replay = QLearningAgent(alpha=0.5, epsilon=0.25, discount=0.99,\n",
    "                       get_legal_actions = lambda s: range(n_actions))\n",
    "\n",
    "# replay = ReplayBuffer(1000)\n",
    "# prioritized = False\n",
    "replay = PrioritizedReplayBuffer(1000)\n",
    "prioritized = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline : eps = 2.9191091959171894e-05 mean reward = 8.1\n",
      "ExpReplay: eps = 2.9191091959171894e-05 mean reward = 7.7\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD8CAYAAAB6paOMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VPW9+P/Xe5bsZIFIWAIkKAIBImBYFLURFXDDpbTiVUGtxY322ttatX5dqtXrr7W3rd6qpRVbq71ira0UsaLWSLEqiyICYQkQIBC27Htm+fz+OIcsEEjIZDLJzPv5eMxj5nzO9p5PJud91s9HjDEopZSKXI5QB6CUUiq0NBEopVSE00SglFIRThOBUkpFOE0ESikV4TQRKKVUhNNEoJRSEU4TgVJKRThNBEopFeFcoQ6gI1JTU01GRkan56+pqSE+Pr7rAurFtC5a0/poTeujWTjUxbp1644YY05rb7pekQgyMjJYu3Ztp+fPy8sjNze36wLqxbQuWtP6aE3ro1k41IWI7O7IdHpqSCmlIpwmAqWUinCaCJRSKsJpIlBKqQiniUAppSKcJgKllIpwmgiUUirChew5AhGZBfwKcAK/M8Y8FapYlOoon9/g8fmp9/ho8Ppp9Ppp8FqffX5Dv4RojDEYY03rN0df4DcGn98ad7RMAIcITkfzy+UQXE7B5XAgAlX1XirrPXh9BoeACIhI07wi1jvHDDsEHA7BaS/fIQan34PD70H8jTh81rv4Pfg8DXj8Qk3pfvbv3IyIwRjBuGMxODECfiMYwIgTv7gQXz14GzBeD+JvAG8jfnFhHE7wNiL2PF4DXj94fIbKBi/VXjf1fgeNXi8erxeX8RHv8uHCi8vfiA8HtRKP+L0YEfziwocTr7jA4QKHG+NwIQ4X4nRhEKxK9eIwXpzGi9s04vI34MSPU/zWHq/xI2IQYxD8iDUnYvwthu1xBvYW7mKVtxxaTW/Pby8L48cblYxxuDHiwDhcGHGB02XXhRsc1v62w/jB78Xj8eD3NmJ8Xozx4feDMT5ruQJgcBhjL9/Qp08S55x7QVB/1yFJBCLiBH4NXAIUAWtEZKkxZnMo4lG9kzGGynovlXUequq9VNV7moYr6z1U1nlp8PqobvBS2+jDbwzYG2ED+A0cPFjPn/d9ToPXT22jl5pGHx6vH6/fT02Dj9pGL1574+/xWRtye+0kUkNfqSIaD0nUkCLVJEs1dSYKB4Y4acCNFx8OvDhx46UPtSRIPdF4SKSGWGkgGi/RNBItHqLwEo2HaBppxGAQXPhJAfwtDuBd+HCJDxc+HBhKTCIO/NaGz35Fi4cYrOW78OKU9vsnvxxgQxD+WEHkMU7c4gvOwouCs9hTsdU1Es5dHdR1hOqIYDJQYIzZCSAirwFXAZoIIlBto5dDlQ0cqW6gst7aqNc0+Civa6S0upGaRh91jV6q6r2U1jZSXF5PTaOXmgYv/na2bU6H0CfGRZzbiYjgcIBg7S1H4SWpbh+JDZtI5zCpjiriHY3E2BvkmGgPMTEe4n2V9PGWEOWvwzjcRPlqifWU4zDeTn1fv8ON3xGFNyoRnysOnyMKvyMKnyMRnyMKryMKr7jx40CMH4fLhdvpwIm1B2oQa6/c4bL2ko2flMYKDA6MODHiwC9O6hzRVDmi8Trc+MSNT5x4icJ/dNjhwidu/A434ozCJT4O79vNgIGDMAYQg8tbZx1h4Lf3ng0OfIjx4XdGY5zRGIcbvzMK43Dj9HsQ/NYesgGHgNMBTgG3Q4iLEmLw4MaL0+XE5XRhHFE0GGfTMhzeelz+RsTptFKhz9rTF7/H2ov2ecB+N75GMD68Dnvv22kdLRhHFH53LH5x4wfAgRHBGAFxHD2swuAAcVhHOk2fBSPClq3bGDlqlLWnjwOwD8eOHhuIvadfV4q1h+FFjA98HvB7rWG/F4zPrk/BONy4XG6crijE6UacDpzisCoKe9kiYOx3EfrHJHfqd3YqQpUIBgN7WwwXAVNCFIvqQl6fnz2ltdQ2+kiIduF0CHvLatlfXk9pTQOlNR489jT7y+vYX15HWa3nhMuLdTuJj3YSF+UiIdpFcpyb80akkhDdPJwY6yYxxkVCtJukWDeJMQ6SvCUkRAmuqv1QtgtKd1nvVQesBZfthsoiMH6oa7FCZzS47JczGpxRkNAPErIgKgF8DdZ7fCrEpVrvrhiISYK4vhCbAp466xSGO9ZahvGD32OVRSficMfgoGe275KXl8fkEDSrENXta2zf1so4Bk3IDXUY3SJUv0Vpo6zVvp2ILAAWAKSlpZGXl9fplVVXVwc0fzjpqrqo8Rj2VPopazAcqvWzv9p6HagxeNvZS3cJ9I8X+sU4GNdX6J/uJjlaSIoS4txCrEuIdUGcS4h2WT8Vp7cWt6eSxMptOD21GK8bqfZBiUGMl/iavcTUHyK27gDRDYdx+htbrdPgoCE6FY87ASMu6mIzqBs6lXJJwSRnUBebhsedbJ3f7qhG+9WkzH71Xvq/0iyS6iJUiaAIGNJiOB3Y33ICY8wiYBFATk6OCaTxp3BoPKqrdKYufH7Dl0XlfL67jA1FFew8Us3m/ZVNp2VEYGjfOEamJ3B5WgIj+vchMcZFTaMXj9cwOCWWQcmxJMa4SIx14xTB4WhrX8BWuhMOb4WSHbBvLexbB+V7Th5kVAIkDoLMHEhMh5Rh4HBCcgakZCDJQ4lxRRFjT54YQH2EM62PZpFUF6FKBGuAESKSCewD5gL/EaJYVBsq6z18uOUQH209TN62w5TWWLu+g5JiGNYvnu9MH8HZw1IYnBLL4ORYYtynsCfdkqcOdv8bNv8NdnxolVW0OGuYNBQGT4TxN0D8aZCeAzHJUF9hnYbx1oPTDcnD7PO3SqlTFZJEYIzxishC4F2s20cXG2M2hSIW1czr8/Pe5oP85fN9rNx2mEafn77xUXztzNOYPqo/U4b3pX+fmPYX1J6KfbDtHdj+Puz80NqYHz3vnpgOU++CAWMhdST0SQt8fUqpkwrZ9SpjzHJgeajWr5rVNHh5fe1efvevXewrr2NAYgw3nTOMy8YNYPyQFJwnO43T7sKPwI5/wq6PoGgdNFY37/EnDoazb4bTp0PmBdbFVaVUt+uJNy6obmCM4aNth3nhox2sLSzD6zdMykjhkSuzuGh0WmAb/7py2LIMdubBlrfBU2vdspeSCSkZ1sZ/1OXWHr9DH25XKtQ0EUQYYwwfFxzhf97bxrrdZQxOjmXBBcO5aHQaZw9LCWzhR7bDhiXw2SJoqLBuvRxzDYz7JgyZZN1iqZTqcTQRRIjdJTX8e0cJv19dz9Z3P2NAYgw/uXos38wZQpQrgL1yY2DD67Du97Dn31bZ6Cuti7vDc/V0j1K9gCaCMFd4pIanV2xl2YZiAJKihUevzGLu5KGdv9PnqJId8Pf/hMJ/QfJQuPhRGHUFpI4IOG6lVPfRRBBmjDGs3V3GU+9s4WBlPfvL64h2Obkz93QuHzeQA1s/5+JpmYGtxO+DT/4XPnzSOv1z5a9g4ny9fVOpXkoTQRipafDyvSXrWbH5YFPZN85O5wczR5KWaN32eWR7gBvr8j3w1ztg98fW3v9lT0PiwMCWqZQKKU0EYeKTHSU89NZGdh2p4YezRnLj1GF4fYa+8V3Uiosx8NWf4e3vW23nXP0CnDVXjwKUCgOaCMLAq5/t5qG/baRfQjQvzs8hd2T/rlnwnk8h9UyrqYY3F8C2f0D6ZPj6b63bQJVSYUETQS9mjOGVT3fz0FubmD6qP89eP4H46C74k5bvhb/daV0EPsrhgplPwpQ7rMSglAobmgh6qXqPj++//iVvf1XM9FH9ef7GiUS7AtxAe+rgHw/AF6+0vu1zzLUw9U4YMjmw5SuleiRNBL1Qo9fPwj99zvv5h7h35khuv2A4LmcAzwJUH7Ke/v3LbVC0BvqeDv/xOvQZYDXuljS464JXSvU4mgh6mUOV9dyzZD3/3lHCT64ey41Th3V+YYUfwxu3QLV9l5E4YM5LMPba5mmiEwILWCnV42ki6EX++EkhP39vG1X1Xh6/akxgSaByP7w+D2qPQL8zYNBEOOs6OOPiLotXKdU7aCLoJR5+ayMvf7KbM/onsOimHCZn9u38wnweeONW65rA3WvgtDO7LlClVK+jiaAXeGv9Pl7+ZDfzzxnGg5dnBdQ2UGxtETwxwOpc++svahJQSmki6On2lNTy4F83cvawFB66Iiuwi8I1R8je8LiVBK56DsbN6bpAlVK9liaCHszj8/Od177AIfCrueM7lwS8DeCKtq4JLLqQ2PoD8M0/Qtbsrg9YKdUraSLooXYcruZvX+zjy73lPHfDRNJT4jqxkA/hj1dD5tegsQZqS9gxfB6naxJQSrWgiaAHWrHpAAv+uA6AK88axGXjOtGo24bXraeDweomEmDOS+w90pfTuyhOpVR40H4Ce5h6j4/Hlm1uGn7oitGnvpBdK+HNb0P/LJi3FHJuhdn/2/r5AKWUsukRQQ/i8xvmLV5NUVkdP5hxJuecnkr/PjGntpDGWlj6Heg7HG59F6LiYPjXghOwUiosaCLoQX6+Yiurd5Vy78yR3H3hGZ1byL+fhbJCmP93KwkopVQ79NRQD/FxwRGe/2gH38xJ567cTp7Fr9wPH/8Ssq6CzAu6NkClVNjSRNADVNZ7WPinz8noF8/DV45BOtPZi99vdRrj98Ilj3V9kEqpsBVQIhCRb4jIJhHxi0jOMeMeEJECEdkqIjNblM+yywpE5P5A1h8ufvPRDspqPTx7/QQSOtOfgDFW43Fbl8OFD2qnMUqpUxLoEcFG4FpgZctCEckC5gJjgFnAcyLiFBEn8GvgUiALuN6eNmIdrKznxVW7uGr8IMYOTurcQj74MWz+G1xwL0z7z64NUCkV9gK6WGyMyQfaOpVxFfCaMaYB2CUiBcDRXk0KjDE77fles6fdfOwCIsVzHxbg9Rm+f8nIU5uxYh+88nU4nG8NnzYKcn+kfQgrpU5ZsO4aGgx82mK4yC4D2HtM+ZS2FiAiC4AFAGlpaeTl5XU6mOrq6oDmD5YDNX5e/bSOcwe72PnVanZ2cD53YwXT/j2vVdm/Rj6Cb+XKE8zRrKfWRahofbSm9dEskuqi3UQgIu8DA9oY9aAx5q0TzdZGmaHtU1GmrQUYYxYBiwBycnJMbm5ue6GeUF5eHoHMHwzltY3M+uW/cDgcPHnD+afWhMTS71jv5/8AvPVw7nc4v09bf6Lj9cS6CCWtj9a0PppFUl20mwiMMZ3pqaQIGNJiOB3Yb38+UXlEefWzPRyorOeVb005tSRQUQTr/wSTvg0XPRS8AJVSESNYt48uBeaKSLSIZAIjgNXAGmCEiGSKSBTWBeWlQYqhx/L4/Lz8SSHnj0jlvBGpHZupdBcc3gorn7aGp303aPEppSJLQNcIROQa4FngNOBtEVlvjJlpjNkkIq9jXQT2AncbY3z2PAuBdwEnsNgYsymgb9ALLf+qmIOVDTx1bXbHZshfBktuaB6ecickDw1OcEqpiBPoXUN/Bf56gnFPAE+0Ub4cWB7IenszYwyLV+1ieGo8XzvztPZn8NTD8nubhwdNgIsfDVZ4SqkIpG0NdbPP95TzZVEFj181BoejA7d6rvwZVO2HG9+EtDEQ2xdcUcEPVCkVMTQRdCOvz8+Ty/NJinVz7cT09meor4TPfgNnzoLTp+szAkqpoNBE0I1+/+9C1u0u41dzxxPfkaYkln4HGqvgaz/UJKCUChptdK6b1Ht8vPrZHnKGpXDV+MHtz/DZb5qbjRh8dvADVEpFLE0E3eQX722jsKSGuy7sQBPTfh/86+dWX8Nf03b5lFLBpYmgG3h8ft5YV8SlYwcwfVRa+zNsWQbVB2Hyt8GpZ++UUsGliaAbfLjlECU1jcw5uwMXiAE+eQ6Sh8HIy4IbmFJKoYmgW/x5XRGpCdFcMKIDzw0UrYW9n8LUO8HhDH5wSqmIp4kgyI5UN/DhlkNcO3EwLmc71V19GP7xAEQnwoQbuydApVTE0xPQQbb8q2K8fsO1E9u5U2jjX+CDx61+hy9/GqL7dE+ASqmIp4kgyJZtKGZE/wRGDUg8fqQx1vMBOz+Cv95hld30N8iY1r1BKqUimiaCIDpYWc+awlLuuejMtif4w5VQ+C/rc78RMP/vkDiw+wJUSin0GkFQvfNVMcbA5dltdBqz+a3mJAAw/f9pElBKhYQmgiBa/tUBzkxL4Iz+x5zv99TBB49Zt4jm/sgqGzGj+wNUSik0EQRNUVkta3aXcvm4Qa1HGAMr/h+UFMCFD0LuffBwGUSdQi9lSinVhTQRBMkLH+3A7XAwJ+eYh8gK3oc1v7M+j5tjvTv0z6CUCh3dAgXB5v2VvPLpHmaNHcDg5NjWI7css97/48/6wJhSqkfQRNDF/H7DPUu+AOCGKUOPHQlblkPW1XCmXhNQSvUMmgi62IrNB9h2sJr7Zo1iyvB+rUfuWwc1h2DU5aEJTiml2qCJoIt9XFBCfJSTb5+fefzI7StAHHDGxd0fmFJKnYAmgi62elcpZ2f0bbtdoe0rYMgUiOvb/YEppdQJaCLoQmU1jWw9WMWUzDY29LWlULze6ntYKaV6EE0EXejvG/YDMCmjjURQuMp6z7ygGyNSSqn2BZQIRORnIrJFRDaIyF9FJLnFuAdEpEBEtorIzBbls+yyAhEJm34Yy2sbeXzZZrIGJjJxaPLxE3z+MrjjYdDE7g9OKaVOItAjgveAscaYbGAb8ACAiGQBc4ExwCzgORFxiogT+DVwKZAFXG9P2+ut212Gx2d46Iqs468P1JZCwXsw+TZwRYUmQKWUOoGAEoExZoUxxmsPfgocfYz2KuA1Y0yDMWYXUABMtl8FxpidxphG4DV72l5vTWEZbqcwoa2jgS9esd7PnNW9QSmlVAd05TWCW4F37M+Dgb0txhXZZScq79WMMXyQf5DxQ5KJcbfxtPDWd2BANgw9p/uDU0qpdrTbH4GIvA+00Y4yDxpj3rKneRDwAq8ena2N6Q1tJx5zgvUuABYApKWlkZeX116oJ1RdXR3Q/O0pKPex/VA9N4+JOm494vdxXtFaigfOpOCjj4IWQ0cFuy56G62P1rQ+mkVSXbSbCIwxJ336SUTmA1cAFxljjm7Ui4AhLSZLB/bbn09Ufux6FwGLAHJyckxubm57oZ5QXl4egczfnuVvfElcVDHf/0YufWLcrUfuXw8rG0mfejXp44IXQ0cFuy56G62P1rQ+mkVSXQR619As4D5gtjGmtsWopcBcEYkWkUxgBLAaWAOMEJFMEYnCuqC8NJAYQq3e42PZhmKuzB50fBJoqIY/zLY+Z5zX/cEppVQHBNpV5f8C0cB7IgLwqTHmDmPMJhF5HdiMdcrobmOMD0BEFgLvAk5gsTFmU4AxhFR+cSW1jT5yR57WeoTfB//d4vJHn7bOrimlVOgFlAiMMWecZNwTwBNtlC8Hlgey3p7CGMP9f/kKgDGDklqPPLCh+fMt76CUUj2Vdl4fgMKSWrYerOLmczMY2u+YHsa+eNVqYO6eryApve0FKKVUD6BNTATgwy2HALhu0pDWI+rKYP2rkD1Xk4BSqsfTRNBJXp+fRSt3clZ6EiPTjumcPn8ZeGphyoLQBKeUUqdAE0EnLd94gAOV9SycPgKHo8VjEw3VsHQhRCfBwPGhC1AppTpIE0EnrSssJSHaxUWj+jcXGgPPTLA+Z80Gaeu5OqWU6lk0EXRS/oEqRg7o0/pooHi91RWlMwouezp0wSml1CnQRNAJxhjyiysZPfCYawO7/2293/MVuGO6PzCllOoETQSdsL+inqp6L6MGJDYXlu6Cd38ECWn68JhSqlfRRNAJ+fsrARg90E4E9ZXwjH1heNw3QhSVUkp1jiaCTsgvthLByAH2qaGDdisZCWkw/aEQRaWUUp2jiaATthyoYmjfOBKi7Qez19tPEd++Uq8NKKV6HU0EnbBxf0XrC8V7PrF6H9NrA0qpXkgTwSl6bfUedpfUkpmaYBV46qwLxf3DoutlpVQE0kRwil5fa/W0OfusQVbB1nfA+CDzghBGpZRSnaeJ4BQVldXxjbPTyRpk3zG0/lWISdKOZ5RSvZYmglNQ3eDlUFUDGanxVkHpLih4H0ZdAY42Oq1XSqleQBPBKSg8UgPA8KOJoHi99T7pthBFpJRSgdNEcAp22YkgIzUe8v8Of74Z4vpB2tjQBqaUUgHQRHAKth+qxiGQ0S8eltxoFZ59C7iiQhuYUkoFQBPBKVi/t5wz0/oQa+qsgr6nw9fuC21QSikVIE0Ep6DgYBVZAxPh0GarYMbjejSglOr1NBF0UIPXR3FlPUP6xsGulVbhgHGhDUoppbqAJoIOWrOrDGNgeGocrF0MGedD8tBQh6WUUgHTRNBB720+QFyUk8t8H0LlPhh/Q6hDUkqpLhFQIhCRx0Vkg4isF5EVIjLILhcReUZECuzxE1vMM19Ettuv+YF+ge6SX2xdH3Dv+9R6kvisuaEOSSmlukSgRwQ/M8ZkG2PGA8uAh+3yS4ER9msB8DyAiPQFHgGmAJOBR0QkJcAYgs7vN2wurrSalTi4GQaepR3TK6XCRkCJwBhT2WIwHjD256uAl43lUyBZRAYCM4H3jDGlxpgy4D1gViAxdIeisjqqG7yMHpAAh7dA/zGhDkkppbqMK9AFiMgTwDygArjQLh4M7G0xWZFddqLytpa7AOtogrS0NPLy8jodY3V1dUDzrzvoBcC1+2Pw1LKl3MmBAJYXSoHWRbjR+mhN66NZJNVFu4lARN4H2upx5UFjzFvGmAeBB0XkAWAh1qmfts6bmJOUH19ozCJgEUBOTo7Jzc1tL9QTysvLI5D5P39vGw7ZzlWj42EzjDr/Wkaln93p5YVSoHURbrQ+WtP6aBZJddFuIjDGXNzBZf0JeBsrERQBQ1qMSwf22+W5x5TndXD5IZNfXMnw0xKIWv9LcMVA/1GhDkkppbpMoHcNjWgxOBvYYn9eCsyz7x6aClQYY4qBd4EZIpJiXySeYZf1SFX1Hq5f9CnvbT7IWYOToGgdjLkWouJDHZpSSnWZQK8RPCUiIwE/sBu4wy5fDlwGFAC1wC0AxphSEXkcWGNP95gxpjTAGILm7Q3FfLKzBIDrBx+E/AoYmB3iqJRSqmsFlAiMMV8/QbkB7j7BuMXA4kDW2108/ubLF+O+eMT6MGJGiKJRSqng0CeLT6K0utH+ZIiq3AMT50G/00Mak1JKdTVNBCdRXGE1N335GbGIpwZSR4Y4IqWU6nqaCE5iX3kdZ6Un8espZVZB6oiTz6CUUr2QJoKTKK6oZ2BSLKx8GvqdAadfFOqQlFKqy2kiOAFjDPvL6xiUHAsVe62LxM6AH8RWSqkeRxPBCew8UkNto49hCT5orIY+bT1crZRSvZ8mghN48u18AIZF2e3q9RkYwmiUUip4NBG049w0q8E5PSJQSoUrTQQnUFLTyHlnpBJVe8gq0CMCpVSY0kRwAgcq6hmYFANVxVaBHhEopcKUJoI2eH1+DlXVM8ZdDO89BPGnQXSfUIellFJBoYmgDYerG/AbuHLXY1ZBmvZIppQKX5oI2rCm0HqS2BljHwVc/j8hjEYppYJLE0EbVm0/TEqcmySXF4ZfqA3NKaXCmiaCNhRX1JPZNwYpKYDkoaEORymlgkoTQRv2ldXxDfkn1JfDsGmhDkcppYJKG885Rr3HR5+yjVxf/QurQPsnVkqFOT0iOMbGfRUMMEeaC1IyQhaLUkp1B00Ex/hw6yFSHVXWgDsOYpJCG5BSSgWZnho6xtjNvyAj+nPwAfcVhjocpZQKOk0ELZXs4NKK16zPMcngig5tPEop1Q301FALDfu+bB5InxS6QJRSqhtpImjh7X+tBaBo+Dch94EQR6OUUt2jSxKBiPxARIyIpNrDIiLPiEiBiGwQkYktpp0vItvt1/yuWH9XqTm8izpiGHjDbyD97FCHo5RS3SLgawQiMgS4BNjTovhSYIT9mgI8D0wRkb7AI0AOYIB1IrLUGFMWaByBqvf4SPUdpq7PQGKdeqCklIocXbHF+wXwQ6wN+1FXAS8by6dAsogMBGYC7xljSu2N/3vArC6IIWAlNY0MkhIa4gaFOhSllOpWASUCEZkN7DPGfHnMqMHA3hbDRXbZicpD7khVA4OkBF9ijwhHKaW6TbunhkTkfaCt7rkeBH4EzGhrtjbKzEnK21rvAmABQFpaGnl5ee2FekLV1dXtzv/VgVq+IxWsq3VSEMC6erqO1EUk0fpoTeujWSTVRbuJwBhzcVvlIjIOyAS+FBGAdOBzEZmMtac/pMXk6cB+uzz3mPK8E6x3EbAIICcnx+Tm5rY1WYfk5eXR3vy1eR/DFsgYN5V+0zq/rp6uI3URSbQ+WtP6aBZJddHpU0PGmK+MMf2NMRnGmAysjfxEY8wBYCkwz757aCpQYYwpBt4FZohIioikYB1NvBv41wicv6QAgIT+mSGORCmlulewnixeDlwGFAC1wC0AxphSEXkcWGNP95gxpjRIMZySrMI/Um/cxAzTB8mUUpGlyxKBfVRw9LMB7j7BdIuBxV213i7h85BRtZZ3XBdyeVRcqKNRSqlupTfMA1QfxIFhZ/ToUEeilFLdThMBQGUxAPUx/UMciFJKdT9NBAA1hwHwxmkiUEpFHk0EYPVNDDjjUkIciFJKdT9NBAB1ViJw99FEoJSKPJoIAG+t1eZdTIImAqVU5NEeyoDG6lJqTSyJcTGhDkUppbqdJgLAU1NGFfEkxbpDHYpSSnU7PTUE+GvLqDSaCJRSkUkTAWDqKqgw8aQmaGf1SqnIo4kAKC05RAXxDOmrzUsopSJPxCeCjfsqSDDVVJo4EqL1kolSKvJEfCJ4+eOdpFDN1LFnhDoUpZQKiYhPBP7K/USLh6GnZ4U6FKWUComITwR9avdYH/rpEYFSKjJFfCLoW28ngr6nhzYQpZQKkYhPBKc1FtEoUZA4ONShKKVUSER8IjjX+xn747PAEfFVoZSKUBG99SuvqGAoBzmQem6oQ1FKqZCJ6ESwpWAHAKkDh4Y4EqWUCp2ITgSVR/YBkNI/PcSRKKVU6ER0ImgsPwBAn1S9UKyPBYUlAAAR3ElEQVSUilwRnQj8VVYiiEoaGOJIlFIqdCI6EThqDuFHID411KEopVTIBJQIRORREdknIuvt12Utxj0gIgUislVEZrYon2WXFYjI/YGsP1DuusNUORLBqf0QKKUiV1ccEfzCGDPefi0HEJEsYC4wBpgFPCciThFxAr8GLgWygOvtabvHO/fDb6c3DaY0FlPpTuu21SulVE8UrHaXrwJeM8Y0ALtEpACYbI8rMMbsBBCR1+xpNwcpjtY+e956rz6EL+40hvr2cChlMkO6ZeVKKdUzdUUiWCgi84C1wPeNMWXAYODTFtMU2WUAe48pn9LWQkVkAbAAIC0tjby8vE4HWF1dTV5eHtNcfXB7q9j21s+prK0jR0rZTL+Alt3bHK0LZdH6aE3ro1kk1UW7iUBE3gcGtDHqQeB54HHA2O8/B24FpI3pDW2fijJtrdcYswhYBJCTk2Nyc3PbC7VN/9hYTP6BTdx7RS4UjIKiNZy5/YWm8aMnnMPA8zq37N4oLy+PztZlONL6aE3ro1kk1UW7icAYc3FHFiQivwWW2YNF0OqMSzqw3/58ovKguOOVzwG4dy7gazxufOzIi4K5eqWU6vECvWuo5Q341wAb7c9LgbkiEi0imcAIYDWwBhghIpkiEoV1QXlpIDGcEp8H0icBUGViyal/nsR++gyBUiqyBXqN4KciMh7r9E4hcDuAMWaTiLyOdRHYC9xtjPEBiMhC4F3ACSw2xmwKMIaO8zZA/9GsHnkvt71dQSXxOBxtncVSSqnIEVAiMMbcdJJxTwBPtFG+HFgeyHo7zdcIzmgKY7KoZAMj+ieEJAylIoHH46GoqIj6+vpQh9IpSUlJ5OfnhzqMDomJiSE9PR23u3PPRAXr9tGeydcIrihqGr0AvH77OSEOSKnwVVRURJ8+fcjIyECk9x15V1VV0adPn1CH0S5jDCUlJRQVFZGZmdmpZURWExPeBnBGUdNgJYK4aGeIA1IqfNXX19OvX79emQR6ExGhX79+AR15RUwiMMbYp4aiqGn04XYK0S5NBEoFkyaB7hFoPUdMImjw+u1TQ9H8eW0RHl+bjy8opVS3ysjI4MiRIyGNIXISwZ514PdS7XVwpLoh1OEopcKM1+sNdQidFjGJIOmPlwCwv9of4kiUUt3llVdeYfLkyYwfP57bb78dn8/H7t27GTFiBEeOHMHv93P++eezYsUKCgsLGTVqFPPnzyc7O5ubbrqJ2traky7/0UcfZcGCBcyYMYN58+bh8/m49957mTRpEtnZ2fzmN78BrKeUL7jgAq655hqysrK444478PuP3xZdffXVnH322YwZM4ZFixYB8OKLL/K9732vaZrf/va3/Nd//VcX1lKk3TUEFJnTAPjn978W4kiUihw//vsmNu+v7NJlZg1K5JErx5xwfH5+PkuWLOHjjz/G7XZz11138eqrrzJv3jzuu+8+7rjjDqZMmUJWVhYzZsygsLCQrVu38uKLLzJt2jRuuukmnnvuOX7wgx+cNI5169axatUqYmNjWbRoEUlJSaxZs4aGhgamTZvGjBkzAFi9ejWbN29m2LBhzJo1izfffJM5c+a0WtbixYvp27cvdXV1TJo0ia9//evMnTuX7OxsfvrTn+J2u3nppZeaEkxXCesjAp//+OsA6yWLWLeTzNT4EESklOouH3zwAevWrWPSpEmMHz+eDz74gJ07dwJw2223UVVVxQsvvMDTTz/dNM+QIUOYNm0aANdddx2rVq1qdz2zZ88mNjYWgBUrVvDyyy8zfvx4pkyZQklJCdu3bwdg8uTJDB8+HKfTyfXXX9/msp955hnOOusspk6dyt69e9m+fTvx8fFMnz6dZcuWsWXLFjweD+PGjQu4floK6yMCj6/50KvE1Z/EURfyzJpqhp8Wr3czKNWNTrbnHizGGObPn89///d/HzeutraWoqIiwGpl9OjzAsduFzqynYiPb96pNMbw7LPPMnPmzFbT5OXltbvsvLw83n//fT755BPi4uLIzc1tuiX0tttu48knn2TUqFHccsst7cZ0qsL6iKBlIojy1fH5AQ8Au0tOft5PKdX7XXTRRbzxxhscOnQIgNLSUnbv3g3Afffdxw033MBjjz3Gt7/97aZ59uzZwyeffALAG2+8wXnnnXdK65w5cybPP/88Ho+1rdm2bRs1NTWAdWpo165d+P1+lixZctyyKyoqSElJIS4uji1btvDpp80t+U+ZMoW9e/fypz/9ieuvv/4Ua6J9YZ0IvC1uEY0xdVT5owH7mQKlVFjLysriJz/5CTNmzCA7O5tLLrmE4uJiPvroI9asWdOUDKKionjppZcAGD16NH/4wx/Izs6mrKyMO++8E4CHH36YpUvbbx/ztttuIysri4kTJzJ27Fhuv/32pruJzjnnHO6//37Gjh1LZmYm11xzTat5Z82ahdfrJTs7m4ceeoipU6e2Gv/Nb36TadOmkZKS0hXV00pYnxpyuxx867xM8r7cgdvjZV+t9XWnnaGd1SsVCa677jquu+6648pb7m2/+eabABQWFuJwOHjhBau/kqqqKuLi4gB47LHH2lz+o48+2mrY4XDw5JNP8uSTTx43bVxcHEuWLDmuvLCwsOnzO++8c8LvsmrVqlZ3D3WlsD4iSIh28dAVWQyOts6zFVZZ5b++YWIIo1JKqY4rLy/nzDPPJDY2losuCk7/KWF9RACAz8t3k1ZCNVQTy9C+cSTGdK6FPqVU+MrIyGDjxo3tT9gJubm5ne7tLDk5mW3btnVtQMcI6yMCADYsIWffywDUmhj6xkeFOCCllOpZwj8R1Jc3fawhhvFDkkMYjFJK9Tzhnwha9FM8dGB/5p0zLITBKKVUzxP+1wi8zYngsTmT4TTtlUwppVqKgCOCFi2NRmkSUCqSFBYWMnbs2KAsOy8vjyuuuAKApUuX8tRTTwVlPd0hAo4I7EQwbBokDg5tLEqpsDR79mxmz54d6jA6LQKOCBrxuOLhluXgjgl1NEqpbub1epualp4zZw61tbU89thjTJo0ibFjx7JgwYKm1gaeeeYZsrKyyM7O5uabbwagpqaGW2+9lUmTJjFhwgTeeuut49bx+9//noULFwJw8803893vfpdzzz2X4cOH88YbbzRN97Of/aypiepHHnkk+F++g8L7iMDbAAUf4PbWhDoSpSLbO/fDga+6dpkDxsGl7Z+Oadm09K233spzzz3HwoULefjhhwG46aabWLZsGVdeeSVPPfUUu3btIjo6mr179wLwxBNPMH36dBYvXkx5eTmTJ0/m4osvPuk6i4uLWbVqFVu2bGH27NnMmTOHFStWsH37dlavXo0xhtmzZ7Ny5UouuOCCwOsiQOF9RFBfCaU7Qh2FUiqEWjYtfeONN7Jq1So+/PBDpkyZwrhx4/jnP//Jpk2bAMjOzuaGG27glVdeweWy9pNXrFjBU089xfjx45taBN2zZ89J13n11VfjcDjIysri4MGDTctZsWIFEyZMYOLEiWzZsqWpiepQC/iIQES+AywEvMDbxpgf2uUPAN8CfMB3jTHv2uWzgF8BTuB3xpjgXWGJ7frGmZRSndCBPfdgaav557vuuou1a9cyZMgQHn300abmnt9++21WrlzJ0qVL+fGPf0x+fj7GGP7yl78wcuTIVss5uoFvS3R0dNPno6edjDE88MAD3H777V311bpMQEcEInIhcBWQbYwZAzxtl2cBc4ExwCzgORFxiogT+DVwKZAFXG9PGxzO8D7zpZRqX8umpf/v//6vqfnn1NRUqqurm87h+/1+9u7dy4UXXshPf/pTKioqqK6uZubMmTz77LNNG/QvvviiU3HMnDmTxYsXU11dDcC+ffuamsgOtUC3lHcCTxljGgCMMUe/1VXAa3b5LhEpACbb4wqMMTsBROQ1e9rNAcahlFJtOtq09O23386IESO48847KSsrY9y4cWRkZDBp0iQAfD4fN954IxUVFRhjuPvuu0lOTuahhx7innvuITs7G2MMGRkZLFu27JTjmDFjBvn5+ZxzzjkAJCQk8Morr9C/f/8u/b6dIYG0zS8i64G3sPb664EfGGPWiMj/Ap8aY16xp3sRONq+6ixjzG12+U3AFGPMwpOtJycnx6xdu7ZzQT6aZL9XdG7+MJOXl9fpxq/CkdZHa11ZH/n5+YwePbpLlhUKVVVVTT2X9QZt1beIrDPG5LQ3b7tHBCLyPjCgjVEP2vOnAFOBScDrIjIcaKt/N0Pbp6LazEQisgBYAJCWlkZeXl57obap39gfUV9XR00n5w831dXVna7LcKT10VpX1kdSUhJVVVVdsqxQ8Pl8vSr++vr6Tv/t2k0ExpgT3iclIncCbxrrsGK1iPiBVKAIGNJi0nRgv/35ROXHrncRsAisI4LO76Xk6l5fC1oXrWl9tNbVRwS9aY/6WL3tiCAmJoYJEyZ0at5Abx/9GzAdQETOBKKAI8BSYK6IRItIJjACWA2sAUaISKaIRGFdUG6//zellFJBE+jF4sXAYhHZCDQC8+2jg00i8jrWRWAvcLcxxgcgIguBd7FuH11sjNkUYAxKqR7KGHPc7Zuq6wXaD3tAicAY0wjceIJxTwBPtFG+HFgeyHqVUj1fTEwMJSUl9OvXT5NBEBljKCkpISam803o6I32SqmgSE9Pp6ioiMOHD4c6lE6pr68PaOPanWJiYkhPT+/0/JoIlFJB4Xa7yczMDHUYnZaXl9fpi6+9TXi3NaSUUqpdmgiUUirCaSJQSqkIF1ATE91FRA4DuwNYRCrW8w1K6+JYWh+taX00C4e6GGaMOa29iXpFIgiUiKztSHsbkUDrojWtj9a0PppFUl3oqSGllIpwmgiUUirCRUoiWBTqAHoQrYvWtD5a0/poFjF1ERHXCJRSSp1YpBwRKKWUOoGwTgQiMktEtopIgYjcH+p4uoOIDBGRD0UkX0Q2ich/2uV9ReQ9Edluv6fY5SIiz9h1tEFEJob2G3Q9u7/sL0RkmT2cKSKf2XWxxG4SHbvZ9CV2XXwmIhmhjDsYRCRZRN4QkS32b+ScSP1tiMj37P+RjSLyfyISE6m/jbBNBCLiBH4NXApkAdeLSFZoo+oWXuD7xpjRWD3H3W1/7/uBD4wxI4AP7GGw6meE/VoAPN/9IQfdfwL5LYb/P+AXdl2UAd+yy78FlBljzgB+YU8Xbn4F/MMYMwo4C6teIu63ISKDge8COcaYsVjN4s8lUn8bxpiwfAHnAO+2GH4AeCDUcYWgHt4CLgG2AgPtsoHAVvvzb4DrW0zfNF04vLB6wfsAqwOlZVjdqB4BXMf+TrD6yTjH/uyyp5NQf4curItEYNex3ykSfxvAYGAv0Nf+Wy8DZkbqbyNsjwho/kMfVWSXRQz78HUC8BmQZowpBrDf+9uThXs9/RL4IeC3h/sB5cYYrz3c8vs21YU9vsKePlwMBw4DL9mnyn4nIvFE4G/DGLMPeBrYAxRj/a3XEaG/jXBOBG31hBExt0iJSALwF+AeY0zlySZtoyws6klErgAOGWPWtSxuY1LTgXHhwAVMBJ43xkwAamg+DdSWsK0P+zrIVUAmMAiIxzoVdqyI+G2EcyIoAoa0GE4H9ocolm4lIm6sJPCqMeZNu/igiAy0xw8EDtnl4VxP04DZIlIIvIZ1euiXQLKIHO2Lo+X3baoLe3wSUNqdAQdZEVBkjPnMHn4DKzFE4m/jYmCXMeawMcYDvAmcS4T+NsI5EawBRth3AURhXQhaGuKYgk6sPgFfBPKNMf/TYtRSYL79eT7WtYOj5fPsO0SmAhVHTxP0dsaYB4wx6caYDKy//z+NMTcAHwJz7MmOrYujdTTHnj5s9vqMMQeAvSIy0i66CKtf8Yj7bWCdEpoqInH2/8zRuojI30bIL1IE8wVcBmwDdgAPhjqebvrO52Edsm4A1tuvy7DOZ34AbLff+9rTC9bdVTuAr7Duogj59whCveQCy+zPw4HVQAHwZyDaLo+xhwvs8cNDHXcQ6mE8sNb+ffwNSInU3wbwY2ALsBH4IxAdqb8NfbJYKaUiXDifGlJKKdUBmgiUUirCaSJQSqkIp4lAKaUinCYCpZSKcJoIlFIqwmkiUEqpCKeJQCmlItz/D6NkI3mj1QYxAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import clear_output\n",
    "from pandas import DataFrame\n",
    "moving_average = lambda x, span=100: DataFrame({'x':np.asarray(x)}).x.ewm(span=span).mean().values\n",
    "\n",
    "rewards_replay, rewards_baseline = [], []\n",
    "\n",
    "for i in range(1000):\n",
    "    rewards_replay.append(play_and_train_with_replay(env, agent_replay, replay, prioritized=prioritized))\n",
    "    rewards_baseline.append(play_and_train_with_replay(env, agent_baseline, replay=None, prioritized=prioritized))\n",
    "    \n",
    "    agent_replay.epsilon *= 0.99\n",
    "    agent_baseline.epsilon *= 0.99\n",
    "    \n",
    "    if i %100 ==0:\n",
    "        clear_output(True)\n",
    "        print('Baseline : eps =', agent_replay.epsilon, 'mean reward =', np.mean(rewards_baseline[-10:]))\n",
    "        print('ExpReplay: eps =', agent_baseline.epsilon, 'mean reward =', np.mean(rewards_replay[-10:]))\n",
    "        plt.plot(moving_average(rewards_replay), label='exp. replay')\n",
    "        plt.plot(moving_average(rewards_baseline), label='baseline')\n",
    "        plt.grid()\n",
    "        plt.legend()\n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Submit to Coursera"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from submit import submit_experience_replay\n",
    "submit_experience_replay(rewards_replay, rewards_baseline, <EMAIL>, <TOKEN>)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### What to expect:\n",
    "\n",
    "Experience replay, if implemented correctly, will improve algorithm's initial convergence a lot, but it shouldn't affect the final performance.\n",
    "\n",
    "### Outro\n",
    "\n",
    "We will use the code you just wrote extensively in the next week of our course. If you're feeling that you need more examples to understand how experience replay works, try using it for binarized state spaces (CartPole or other __[classic control envs](https://gym.openai.com/envs/#classic_control)__).\n",
    "\n",
    "__Next week__ we're gonna explore how q-learning and similar algorithms can be applied for large state spaces, with deep learning models to approximate the Q function.\n",
    "\n",
    "However, __the code you've written__ for this week is already capable of solving many RL problems, and as an added benifit - it is very easy to detach. You can use Q-learning, SARSA and Experience Replay for any RL problems you want to solve - just thow 'em into a file and import the stuff you need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}