week01_intro/project_starter_evolution_strategies.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Project :: Evolution Strategies\n",
    "\n",
    "![img](https://t4.ftcdn.net/jpg/00/17/46/81/240_F_17468143_wY3hsHyfNYoMdG9BlC56HI4JA7pNu63h.jpg)\n",
    "\n",
    "Remember the idea behind Evolution Strategies? Here's a neat [blog post](https://blog.openai.com/evolution-strategies/) about 'em.\n",
    "\n",
    "Can you reproduce their success? You will have to implement evolutionary strategies and see how they work.\n",
    "\n",
    "This project is optional; has several milestones each worth a number of points [and swag].\n",
    "\n",
    "__Milestones:__\n",
    "* [10pts] Basic prototype of evolutionary strategies that works in one thread on CartPole\n",
    "* [+5pts] Modify the code to make them work in parallel\n",
    "* [+5pts] if you can run ES distributedly on at least two PCs\n",
    "* [+10pts] Apply ES to play Atari Pong at least better than random\n",
    "* [++] Additional points for all kinds of cool stuff besides milestones\n",
    "\n",
    "__Rules:__\n",
    "\n",
    "* This is __not a mandatory assignment__, but it's a way to learn some cool things if you're getting bored with default assignments.\n",
    "* Once you decided to take on this project, please tell any of course staff members so that we can help ypu if you get stuck.\n",
    "* There's a default implementation of ES in this [openai repo](https://github.com/openai/evolution-strategies-starter). It's okay to look there if you get stuck or want to compare your solutions, but each copy-pasted chunk of code should be understood thoroughly. We'll test that with questions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tips on implementation\n",
    "\n",
    "* It would be very convenient later if you implemented a function that takes policy weights, generates a session and returns policy changes -- so that you could then run a bunch of them in parallel.\n",
    "\n",
    "* The simplest way you can do multiprocessing is to use [joblib](https://www.google.com/search?client=ubuntu&channel=fs&q=joblib&ie=utf-8&oe=utf-8)\n",
    "\n",
    "* For joblib, make sure random variables are independent in each job. Simply add `np.random.seed()` at the beginning of your \"job\" function.\n",
    "\n",
    "Later once you got distributed, you may need a storage that gathers gradients from all workers. In such case we recommend [Redis](https://redis.io/) due to it's simplicity.\n",
    "\n",
    "Here's a speed-optimized saver/loader to store numpy arrays in Redis as strings.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import joblib\n",
    "from six import BytesIO\n",
    "\n",
    "\n",
    "def dumps(data):\n",
    "    \"\"\"converts whatever to string\"\"\"\n",
    "    s = BytesIO()\n",
    "    joblib.dump(data, s)\n",
    "    return s.getvalue()\n",
    "\n",
    "\n",
    "def loads(self, string):\n",
    "    \"\"\"converts string to whatever was dumps'ed in it\"\"\"\n",
    "    return joblib.load(BytesIO(string))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tips on Atari games\n",
    "* There's all the pre-processing and tuning done for you in the code below\n",
    "    * Images rescaled to 42x42 to speed up computation\n",
    "    * We use last 4 frames as observations to account for ball velocity\n",
    "    * The code below requires ```pip install Image``` and ```pip install gym[atari]``` \n",
    "    * You may also need some dependencies for gym[atari] - google \"gym install all\" dependencies or use our pre-built environment.\n",
    "* The recommended agent architecture is a convolutional neural network. Dense network will also do.\n",
    "\n",
    "\n",
    "May the force be with you!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pong import make_pong\n",
    "import numpy as np\n",
    "\n",
    "env = make_pong()\n",
    "print(env.action_space)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the initial state\n",
    "s = env.reset()\n",
    "print(s.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "# plot first observation. Only one frame\n",
    "plt.imshow(s.swapaxes(1, 2).reshape(-1, s.shape[-1]).T)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# next frame\n",
    "new_s, r, done, _ = env.step(env.action_space.sample())\n",
    "plt.imshow(new_s.swapaxes(1, 2).reshape(-1, s.shape[-1]).T)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# after 10 frames\n",
    "for _ in range(10):\n",
    "    new_s, r, done, _ = env.step(env.action_space.sample())\n",
    "\n",
    "plt.imshow(new_s.swapaxes(1, 2).reshape(-1, s.shape[-1]).T, vmin=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<YOUR CODE: tons of it here or elsewhere>"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}