02_MAB_movielens_egreedy.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "02_MAB_movielens_egreedy.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyMMuGz4t+ALrQS7f44dHm9X",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/pstanisl/mlprague-2021/blob/main/02_MAB_movielens_egreedy.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P5Ci5jh6pTY5"
      },
      "source": [
        "# Multi-Arm Bandits - Epsilon Greedy\n",
        "\n",
        "Epsilon-Greedy is a simple method to balance exploration and exploitation by choosing between exploration and exploitation randomly.\n",
        "\n",
        "#### Installation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wF7CuHYDTCFW"
      },
      "source": [
        "!rm -f ./utils.py\n",
        "!wget --no-check-certificate --no-cache --no-cookies \\\n",
        "    https://raw.githubusercontent.com/pstanisl/mlprague-2021/main/utils.py \\\n",
        "    -O ./utils.py"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1CpGY1RFTEK2"
      },
      "source": [
        "##### Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8-kc2t0LpSqd"
      },
      "source": [
        "from tqdm.notebook import trange\n",
        "from typing import Dict, List, Text, Tuple\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import random\n",
        "import seaborn as sns\n",
        "import zipfile\n",
        "\n",
        "from utils import load_movielens_data\n",
        "from utils import plot_accuracy, plot_actions, plot_cumsum, plot_regret\n",
        "\n",
        "# Apply the default theme\n",
        "sns.set_theme()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ud80Y6jPpnPI"
      },
      "source": [
        "#### Downloading the [MovieLens](https://grouplens.org/datasets/movielens/) (100K) dataset.\n",
        "\n",
        "**Dataset info**\n",
        "\n",
        "MovieLens data sets were collected by the GroupLens Research Project\n",
        "at the University of Minnesota.\n",
        "\n",
        "This data set consists of:\n",
        "* 100,000 ratings (1-5) from 943 users on 1682 movies.\n",
        "* Each user has rated at least 20 movies.\n",
        "* Simple demographic info for the users (age, gender, occupation, zip)\n",
        "\n",
        "The data was collected through the MovieLens web site\n",
        "(movielens.umn.edu) during the seven-month period from September 19th,\n",
        "1997 through April 22nd, 1998. This data has been cleaned up - users\n",
        "who had less than 20 ratings or did not have complete demographic\n",
        "information were removed from this data set. Detailed descriptions of\n",
        "the data file can be found at the end of this file.\n",
        "\n",
        "Neither the University of Minnesota nor any of the researchers\n",
        "involved can guarantee the correctness of the data, its suitability\n",
        "for any particular purpose, or the validity of results based on the\n",
        "use of the data set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PeTC-7KaplPj"
      },
      "source": [
        "print(\"Downloading movielens data...\")\n",
        "\n",
        "!wget --no-check-certificate \\\n",
        "    http://files.grouplens.org/datasets/movielens/ml-100k.zip \\\n",
        "    -O ./movielens.zip\n",
        "\n",
        "zip_ref = zipfile.ZipFile('movielens.zip', \"r\")\n",
        "zip_ref.extractall()\n",
        "\n",
        "print(\"Done. Dataset contains:\")\n",
        "print(zip_ref.read('ml-100k/u.info').decode())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hRwiL0Mrigc_"
      },
      "source": [
        "!head ./'ml-100k/u.data'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8AqouXMKTe7J"
      },
      "source": [
        "#### Parameters -- Feel Free to Play Around"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nQRLMMhUTcRc"
      },
      "source": [
        "RANK_K = 20 # @param {type:\"integer\"}\n",
        "NUM_ACTIONS = 20 # @param {type:\"integer\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vjg4v4StqMlm"
      },
      "source": [
        "## Environment\n",
        "\n",
        "Implementation of the environment uses **MovieLens 100K dataset**. As described above, the dataset contains 100000 ratings from 943 users and 1682 movies. The environment can consider only the first $n$ of the dataset's movies. It can be set-up by `num_actions`. The number of \"known\" movies for the environment is equal to actions/arms.\n",
        "\n",
        "> Users without a rating (after selecting first $n$ movies) are removed from the environment.\n",
        "\n",
        "> In every step, the batch of users will be selected randomly.\n",
        "\n",
        "<br/>\n",
        "\n",
        "**TASK** Add code into `_observe` and `step` methods, where\n",
        "\n",
        "1. `_observe` will randomly generate sample (`batch_size`) of users \n",
        "1. `step` will return `reward` from provided batch of actions\n",
        "\n",
        "Hint:\n",
        "\n",
        "* You can use the `sample` method from `random` package\n",
        "* `self._approx_ratings_matrix[i, j]` - `i` is the users index and `j` is the movie index"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wmu1KUCUqHmi"
      },
      "source": [
        "class MovielensEnvironment(object):\n",
        "  def __init__(\n",
        "      self, \n",
        "      data_dir: Text,\n",
        "      rank_k: int, \n",
        "      batch_size: int = 1,\n",
        "      num_movies: int = 20\n",
        "  ):\n",
        "    \"\"\"Initializes the MovieLens Bandit environment.\n",
        "\n",
        "    Args:\n",
        "      data_dir: (string) Directory where the data lies (in text form).\n",
        "          The file is a csv with rows containing:\n",
        "          user id | item id | rating | timestamp\n",
        "      rank_k : (int) Which rank to use in the matrix factorization.\n",
        "      batch_size: (int) Number of observations generated per call.\n",
        "      num_movies: (int) Only the first `num_movies` movies will be used by the\n",
        "        environment. The rest is cut out from the data.\n",
        "    \"\"\"\n",
        "    self._num_actions = num_movies\n",
        "    self._batch_size = batch_size\n",
        "    self._context_dim = rank_k\n",
        "\n",
        "    # Compute the matrix factorization.\n",
        "    self._data_matrix = load_movielens_data(data_dir)\n",
        "    # Keep only the first items.\n",
        "    self._data_matrix = self._data_matrix[:, :num_movies]\n",
        "    # Filter the users with no iterm rated.\n",
        "    nonzero_users = list(np.nonzero(np.sum(self._data_matrix, axis=1) > 0.0)[0])\n",
        "    self._data_matrix = self._data_matrix[nonzero_users, :]\n",
        "    self._effective_num_users = len(nonzero_users)\n",
        "\n",
        "    # Compute the SVD.\n",
        "    u, s, vh = np.linalg.svd(self._data_matrix, full_matrices=False)\n",
        "    # Keep only the largest singular values.\n",
        "    self._u_hat = u[:, :rank_k] * np.sqrt(s[:rank_k])\n",
        "    self._v_hat = np.transpose(\n",
        "        np.transpose(vh[:rank_k, :]) * np.sqrt(s[:rank_k]))\n",
        "    self._approx_ratings_matrix = np.matmul(self._u_hat, self._v_hat)\n",
        "    # Currently generated sample of users\n",
        "    self._current_users = np.zeros(batch_size)\n",
        "    # Previously generated sample of users\n",
        "    self._previous_users = np.zeros(batch_size)\n",
        "\n",
        "    self._optimal_action_table = np.argmax(\n",
        "        self._approx_ratings_matrix, axis=1)\n",
        "    self._optimal_reward_table = np.max(\n",
        "        self._approx_ratings_matrix, axis=1)\n",
        "    \n",
        "    self.reset()\n",
        "\n",
        "  @property\n",
        "  def batch_size(self):\n",
        "    return self._batch_size\n",
        "\n",
        "  @property\n",
        "  def best_action(self) -> int:\n",
        "    return np.argmax(np.sum(env._data_matrix, axis=0))\n",
        "\n",
        "  @property\n",
        "  def n_actions(self) -> int:\n",
        "    return self._data_matrix.shape[1]\n",
        "\n",
        "  def reset(self):\n",
        "    return self._observe()\n",
        "\n",
        "  def _observe(self) -> np.ndarray:\n",
        "    \"\"\"Returns the u vectors of a random sample of users.\"\"\"\n",
        "    \n",
        "    sampled_users = # YOUR CODE HERE\n",
        "    \n",
        "    self._previous_users = self._current_users\n",
        "    self._current_users = sampled_users\n",
        "    batched_observations = self._u_hat[sampled_users]\n",
        "    return batched_observations\n",
        "\n",
        "  def step(self, action: List[int]) -> Tuple[int, float]:\n",
        "    \"\"\"Computes the reward for the input actions.\"\"\"\n",
        "    rewards = []\n",
        "\n",
        "    # YOUR CODE GOES HERE\n",
        "    \n",
        "\n",
        "    # END OF YOUR CODE\n",
        "\n",
        "    # Generate a new users\n",
        "    self._observe()\n",
        "    return np.array(rewards)\n",
        "\n",
        "  def compute_optimal_action(self):\n",
        "    return self._optimal_action_table[self._previous_users]\n",
        "\n",
        "  def compute_optimal_reward(self):\n",
        "    return self._optimal_reward_table[self._previous_users]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6BxSt_kfkz-N"
      },
      "source": [
        "Now we are equipped to initialize our environment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nofEwdPVq1K9"
      },
      "source": [
        "env = MovielensEnvironment(\n",
        "    './ml-100k/u.data', rank_k=RANK_K, batch_size=1, num_movies=NUM_ACTIONS)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j-JYDxYrkcVi"
      },
      "source": [
        "Below we can check what this environment produces."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UjyXqeqt__Aj"
      },
      "source": [
        "env.reset()\n",
        "\n",
        "action = np.zeros(1, dtype=np.int32) # Create action=0 - first movie\n",
        "reward = env.step(action)\n",
        "\n",
        "print(f'For users={env._previous_users}, we selected action={action} (optimal={env.compute_optimal_action()})')\n",
        "print(f'For users={env._previous_users}, we received reward={reward} (optimal={env.compute_optimal_reward()})')\n",
        "\n",
        "# Example of output\n",
        "#For users=[339], we selected action=[0] (optimal=[13])\n",
        "#For users=[339], we received reward=[1.83844185e-14] (optimal=[5.])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ynALMHf_3Uow"
      },
      "source": [
        "## Policy\n",
        "\n",
        "The $\\epsilon-greedy$ algorithm takes the best action most of the time, but does random exploration occasionally (with probability $\\epsilon$). The action value is estimated according to the experience by averaging the rewards associated with the target action that we have observed so far\n",
        "\n",
        "$$\n",
        "\\hat{Q}_{t}(a) = \\frac{1}{N_{t}(a)}\\sum_{\\tau=1}^{t}r_{\\tau, a}\n",
        "$$\n",
        "\n",
        "where $N_{t}(a)$ is how many times the action a has been selected so far.\n",
        "\n",
        "According to the $\\epsilon-greedy$ algorithm, with a small probability $\\epsilon$ we take a random action, but otherwise (which should be the most of the time, probability $1 - \\epsilon$) we pick the best action that we have learned so far\n",
        "\n",
        "$$\n",
        "\\hat{a}_{t}^{\\ast} = argmax \\hat{Q}_{t}(a).\n",
        "$$\n",
        "\n",
        "<br/>\n",
        "\n",
        "**TASK**: Fill in a missing code for generating batch of actions, where\n",
        "\n",
        "* with probability $1 - \\epsilon$ will be selected current best action,\n",
        "* with probability $\\epsilon$ will be the action selected randomly."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K9pcEDmZ259K"
      },
      "source": [
        "class EpsilonGreedyPolicy(object):\n",
        "  def __init__(self, epsilon: float, values: List[float], batch_size: int = 1):\n",
        "    self._epsilon = epsilon\n",
        "    self._Q = values\n",
        "\n",
        "    self._batch_size = batch_size\n",
        "\n",
        "  def action(self) -> int:\n",
        "    val_max = max(self._Q)\n",
        "\n",
        "    ret = []\n",
        "    \n",
        "    # YOUR CODE GOES HERE\n",
        "    \n",
        "\n",
        "\n",
        "    # END OF YOUR CODE\n",
        "    \n",
        "    return ret"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RbQdE3nga6kJ"
      },
      "source": [
        "Below we can check what this policy produces."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sMyypwOQq-T0"
      },
      "source": [
        "p = EpsilonGreedyPolicy(epsilon=0.3, values=[1, 2, 3], batch_size=10)\n",
        "print(f'E-Greedy - action:{p.action()}')\n",
        "\n",
        "# Expected output\n",
        "# E-Greedy - action:[2, 2, 2, 2, 2, 2, 1, 2, 2, 2]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q6lLvHMq4wGp"
      },
      "source": [
        "## Agent\n",
        "\n",
        "As was mentioned above, the $\\epsilon-greedy$ algorithm takes the best action most of the time, but does random exploration occasionally. The action value is estimated according to the past experience by averaging the rewards associated with the target action a that we have observed so far (up to the current time step $t$)\n",
        "\n",
        "$$\n",
        "\\hat{Q}_{t}(a) = \\frac{1}{N_{t}(a)}\\sum_{\\tau=1}^{t}r_{\\tau, a}\n",
        "$$\n",
        "\n",
        "where $N_{t}(a)$ is how many times the action a has been selected so far.\n",
        "\n",
        "<br/>\n",
        "\n",
        "**TASK**: Fill in a missing code for updating the action value estimations.\n",
        "1. initialize `Q` and `N` in `reset` method\n",
        "1. update $N_{t}(a)$\n",
        "1. update $\\hat{Q}_{t}(a)$ (`self._values`) -> $\\frac{n - 1}{n} * Q + \\frac{1}{n} * r$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bH7WxMem3wti"
      },
      "source": [
        "class EpsilonGreedyAgent(object):\n",
        "  \n",
        "  def __init__(self, n: int, epsilon: float = 0.1, batch_size: int = 1):\n",
        "    self._epsilon = epsilon\n",
        "    self._n = n\n",
        "    self._batch_size = batch_size\n",
        "\n",
        "    self.reset()\n",
        "\n",
        "    self.policy = EpsilonGreedyPolicy(\n",
        "        self._epsilon, self._values, batch_size=batch_size)\n",
        "\n",
        "  def reset(self):\n",
        "    self._counts = # YOUR CODE HERE # N_t\n",
        "    self._values = # YOUR CODE HERE # Q_t\n",
        "\n",
        "  def train(self, experience: Dict[str, float]):\n",
        "    \"\"\"Update policy parameters.\n",
        "\n",
        "    Args:\n",
        "      experience: dictionary with a single action and reward\n",
        "    \"\"\"\n",
        "    action = experience['action']\n",
        "    reward = experience['reward']\n",
        "\n",
        "    # YOUR CODE GOES HERE\n",
        "    \n",
        "\n",
        "\n",
        "    # END OF YOUR CODE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WSlZSLqra7-P"
      },
      "source": [
        "Below we can check how the training affects rewards estimation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c73Skpjc5CXU"
      },
      "source": [
        "ag = EpsilonGreedyAgent(3)\n",
        "\n",
        "experience = {'action': 2, 'reward': 1}\n",
        "ag.train(experience)\n",
        "print(f'Q={ag._values}')\n",
        "\n",
        "experience = {'action': 2, 'reward': 0}\n",
        "ag.train(experience)\n",
        "print(f'Q={ag._values}')\n",
        "\n",
        "# Expected output\n",
        "#Q=[0.0, 0.0, 1.0]\n",
        "#Q=[0.0, 0.0, 0.5]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XwlrkZHM6OFW"
      },
      "source": [
        "## Training\n",
        "\n",
        "Now we put together all the components that we introduced above: the environment, the policy, and the agent. We run the policy on the environment and output training data.\n",
        "\n",
        "<br />\n",
        "\n",
        "**TASK**: Add missing code for computing regret (not cumulative)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eeSdkQYa6BDg"
      },
      "source": [
        "def run(environment: object, agent: object, trials=100):\n",
        "  trajectory = []\n",
        "\n",
        "  experience = {\n",
        "      'trial': 0, \n",
        "      'action': -1, \n",
        "      'observation': 0, \n",
        "      'reward': 0,\n",
        "      'regret': -1\n",
        "  }\n",
        "\n",
        "  for i in range(trials):\n",
        "    experience['trial'] = i + 1\n",
        "    actions = agent.policy.action()\n",
        "    rewards = environment.step(actions)\n",
        "    optimal_rewards = environment.compute_optimal_reward()\n",
        "\n",
        "    for action, reward, optimal_reward in zip(actions, rewards, optimal_rewards):\n",
        "      experience['action'] = action\n",
        "      experience['reward'] = reward\n",
        "      \n",
        "      experience['regret'] = # YOUR CODE HERE\n",
        "\n",
        "      agent.train(experience)\n",
        "\n",
        "      trajectory.append(experience.copy())\n",
        "    \n",
        "  df_trajectory = pd.DataFrame.from_dict(trajectory)\n",
        "  df_cumsum = df_trajectory.groupby('action')['reward'].cumsum()\n",
        "  df_trajectory = df_trajectory.assign(cum_sum=df_trajectory['reward'].cumsum())\n",
        "  df_trajectory = df_trajectory.assign(action_cum_sum=df_cumsum)\n",
        "\n",
        "  return df_trajectory.astype({'action': 'int32', 'trial': 'int32'})"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KIHlbfjtU7y6"
      },
      "source": [
        "Down below is the code for creating all necessary instances. We have here a few parameters we can play with. `num_iterations` specifies how many times we run the trainer loop, `batch_size` defines how many actions are generated through one step and `epsilon` defines the probability of randomly selected action."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sQugKS6k6ShO"
      },
      "source": [
        "batch_size =   32 # @param {type:\"integer\"}\n",
        "epsilon = 0.1 # @param {type: \"number\"}\n",
        "num_iterations =   150 # @param {type:\"integer\"}\n",
        "\n",
        "environment = MovielensEnvironment(\n",
        "    './ml-100k/u.data', \n",
        "    rank_k=RANK_K, \n",
        "    batch_size=batch_size, \n",
        "    num_movies=NUM_ACTIONS)\n",
        "step = environment.reset()\n",
        "\n",
        "agent = EpsilonGreedyAgent(\n",
        "    environment.n_actions, \n",
        "    epsilon=epsilon, \n",
        "    batch_size=environment.batch_size)\n",
        "experience = {'action': [-1], 'reward': [0]}\n",
        "\n",
        "df_trajectory = run(environment, agent, trials=num_iterations)\n",
        "\n",
        "print(f'\\Q={agent._values}')\n",
        "print(f'N={agent._counts}')\n",
        "print(\n",
        "  f'best action={np.argmax(agent._values)} '\n",
        "  f'(environment best_action={environment.best_action})'\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XLHQDuOIUwTD"
      },
      "source": [
        "Now let's see the result. After running the last code snippet, the resulting plot (hopefully) shows that the average regret is going down as the agent is trained and the policy gets better in figuring out what the right action is, given the observation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S7P3uSGjV0Cc"
      },
      "source": [
        "params = {\n",
        "  'algorithm': 'E-Greedy',\n",
        "  'best_action': environment.best_action\n",
        "}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZbJN-WND0_jK"
      },
      "source": [
        "plot_regret(df_trajectory.groupby('trial').mean()['regret'], params)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Cv8nTvSKkpp"
      },
      "source": [
        "Let's see the cumulative reward of the actions during the experiment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vFoLhk7O7oKs"
      },
      "source": [
        "plot_cumsum(df_trajectory.copy(), params, show_actions=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XHyfe0J1BLPW"
      },
      "source": [
        "#### Multiple runs"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eiaxbdId-zlb"
      },
      "source": [
        "def experiment(epsilons: List[float], epochs: int = 1, trials: int = 10, batch_size: int = 1):\n",
        "  trajectories = []\n",
        "  \n",
        "  environment = MovielensEnvironment(\n",
        "      './ml-100k/u.data', \n",
        "      rank_k=RANK_K, \n",
        "      batch_size=batch_size, \n",
        "      num_movies=NUM_ACTIONS)\n",
        "  params = {'best_action': environment.best_action}\n",
        "  \n",
        "  for epsilon in epsilons:\n",
        "    for epoch in trange(epochs):\n",
        "      step = environment.reset()\n",
        "      agent = EpsilonGreedyAgent(\n",
        "          environment.n_actions, \n",
        "          epsilon=epsilon, \n",
        "          batch_size=environment.batch_size)\n",
        "      \n",
        "      df = run(environment, agent, trials=trials)\n",
        "      df['epoch'] = epoch + 1\n",
        "      df['epsilon'] = epsilon\n",
        "\n",
        "      trajectories.append(df)\n",
        "\n",
        "  df_trajectory = pd.concat(trajectories, ignore_index=True)\n",
        "\n",
        "  return df_trajectory.astype({'action': 'int32', 'trial': 'int32'}), params"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U14AZz0sXa10"
      },
      "source": [
        "Compared to a single run, we have one extra parameter. `epochs` controls the number of independent runs of the training loop. We can even test multiple `epsilons`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EqthUCJdBcVs"
      },
      "source": [
        "batch_size =   256 # @param {type:\"integer\"}\n",
        "epochs =  25# @param {type: \"integer\"}\n",
        "num_iterations =   150 # @param {type:\"integer\"}\n",
        "\n",
        "df_trajectory, params = experiment(\n",
        "    epsilons=[0.1], \n",
        "    epochs=epochs, \n",
        "    trials=num_iterations, \n",
        "    batch_size=batch_size)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oANiGLamX1kS"
      },
      "source": [
        "Now let's see the average results after running multiple runs. The resulting regret plot (hopefully) shows that the average regret is going down as the agent is trained and the policy gets better in figuring out what the right action is, given the observation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FbOieIuVVruP"
      },
      "source": [
        "params['algorithm'] = 'E-Greedy'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "twKjMjeC1qaW"
      },
      "source": [
        "plot_regret(df_trajectory.groupby('trial').mean()['regret'], params)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OgufnZb7Dbxo"
      },
      "source": [
        "On the other graph, we can see the accuracy of the environment's best action.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o1HJozT0Bf4W"
      },
      "source": [
        "plot_accuracy(df_trajectory, params)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ll-maW27Es1F"
      },
      "source": [
        "Let's see the cumulative reward of the actions during the experiment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PnW7DkA5CB8x"
      },
      "source": [
        "plot_cumsum(df_trajectory[df_trajectory.epsilon == 0.1], params, show_actions=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tQ2IK_5JEbHK"
      },
      "source": [
        "Let's see the selection's rate of the actions during the experiment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mwPGIsHcVnwv"
      },
      "source": [
        "plot_actions(df_trajectory[df_trajectory.epsilon == 0.1], params)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Yu4YEiJNXnUq"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}