HW6/Practical/my_DQN_DRQN.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Teu0MpvtAmY0"
      },
      "source": [
        "# CE-40719: Deep Learning\n",
        "## HW6 - Deep Reinforcement Learning\n",
        "\n",
        "(40 points)\n",
        "\n",
        "#### Name: \n",
        "#### Student No.: \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jNnkTyAeBFY3"
      },
      "source": [
        "In this assignment, we are going to design an agent to play Atari game. \n",
        "we will use a wrapper to change MDP problem to POMDP, as a result, we are able to investigate the efficiency of using memory to solve problems in a partial observability setting.\n",
        "For this reseaon, we use a Deep Q-Network model as memory less architecture and a [DRQN](https://arxiv.org/abs/1507.06527) as a memoryful agent to play game Pong(`PongNoFrameskip-v4` environment of [gym](https://gym.openai.com/) library).\n",
        "In this environment, the observation is an RGB image of the screen, which is an array of shape (210, 160, 3). And action space is 6.\n",
        "We will train the model for 200,000 steps and should take approximately 2 hour.\n",
        "\n",
        "At the end, you should be able to conclude effectiveness of recurrent memory to cancel out noisy observation.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F3oJ01LNAkqh"
      },
      "source": [
        "![Pong](https://cdn-images-1.medium.com/max/800/1*UHYJE7lF8IDZS_U5SsAFUQ.gif)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QowF3gQ9-NK5"
      },
      "source": [
        " ### 1. Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lUeOVRq_BRWx"
      },
      "source": [
        "if you use google colab to train your network, mount to google drive would be necessary "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hrajUUsIBPRV"
      },
      "outputs": [],
      "source": [
        "from google.colab import drive\n",
        "drive._mount('/content/drive')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fLqyJFdM-RuM"
      },
      "source": [
        "First, we need to install `stable-baselines`. This library  is a set of improved implementations of Reinforcement Learning (RL) algorithms based on OpenAI Baselines. We will use some of **wrappers** of this library. Wrappers will allow us to add functionality to environments, such as modifying observations and rewards to be fed to our agent. It is common in reinforcement learning to preprocess observations in order to make them more easy to learn from. \n",
        "\n",
        "- For linux based Operating Systems or google colab run cell below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ThfMQQMF4wuB",
        "outputId": "86d52322-d09a-43e5-b55b-c8909b42f23f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\r0% [Working]\r            \rGet:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]\n",
            "\r0% [Connecting to archive.ubuntu.com] [1 InRelease 14.2 kB/88.7 kB 16%] [Connec\r                                                                               \rGet:2 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic InRelease [15.9 kB]\n",
            "\r0% [Connecting to archive.ubuntu.com] [1 InRelease 31.5 kB/88.7 kB 36%] [Connec\r0% [Connecting to archive.ubuntu.com] [Waiting for headers] [2 InRelease 15.9 k\r                                                                               \rGet:3 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ InRelease [3,626 B]\n",
            "\r0% [Connecting to archive.ubuntu.com] [Connecting to ppa.launchpad.net (91.189.\r0% [1 InRelease gpgv 88.7 kB] [Connecting to archive.ubuntu.com] [Connecting to\r                                                                               \rHit:4 http://ppa.launchpad.net/cran/libgit2/ubuntu bionic InRelease\n",
            "\r0% [1 InRelease gpgv 88.7 kB] [Connecting to archive.ubuntu.com] [Connecting to\r                                                                               \rGet:5 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic InRelease [15.9 kB]\n",
            "\r0% [1 InRelease gpgv 88.7 kB] [Waiting for headers] [5 InRelease 2,572 B/15.9 k\r                                                                               \rHit:6 http://archive.ubuntu.com/ubuntu bionic InRelease\n",
            "\r0% [1 InRelease gpgv 88.7 kB] [Waiting for headers] [5 InRelease 8,364 B/15.9 k\r0% [1 InRelease gpgv 88.7 kB] [Waiting for headers] [Connecting to ppa.launchpa\r                                                                               \rGet:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]\n",
            "\r0% [1 InRelease gpgv 88.7 kB] [7 InRelease 14.2 kB/88.7 kB 16%] [Connecting to \r                                                                               \rGet:8 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease [21.3 kB]\n",
            "Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease\n",
            "Ign:10 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease\n",
            "Get:11 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [26.8 kB]\n",
            "Get:12 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [716 kB]\n",
            "Get:13 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2,489 kB]\n",
            "Get:14 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,459 kB]\n",
            "Ign:15 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease\n",
            "Hit:16 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release\n",
            "Hit:17 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release\n",
            "Get:18 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic/main Sources [1,822 kB]\n",
            "Get:19 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic/main amd64 Packages [934 kB]\n",
            "Get:20 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic/main amd64 Packages [45.3 kB]\n",
            "Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [34.5 kB]\n",
            "Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [749 kB]\n",
            "Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2,238 kB]\n",
            "Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2,929 kB]\n",
            "Get:25 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic/main amd64 Packages [44.7 kB]\n",
            "Fetched 13.7 MB in 2s (8,936 kB/s)\n",
            "Reading package lists... Done\n",
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "zlib1g-dev is already the newest version (1:1.2.11.dfsg-0ubuntu2).\n",
            "zlib1g-dev set to manually installed.\n",
            "libopenmpi-dev is already the newest version (2.1.1-8).\n",
            "cmake is already the newest version (3.10.2-1ubuntu2.18.04.2).\n",
            "0 upgraded, 0 newly installed, 0 to remove and 61 not upgraded.\n",
            "Collecting stable-baselines[mpi]==2.8.0\n",
            "  Downloading stable_baselines-2.8.0-py3-none-any.whl (222 kB)\n",
            "\u001b[K     |████████████████████████████████| 222 kB 12.3 MB/s \n",
            "\u001b[?25hRequirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]==2.8.0) (3.2.2)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]==2.8.0) (1.4.1)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]==2.8.0) (1.1.0)\n",
            "Requirement already satisfied: opencv-python in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]==2.8.0) (4.1.2.30)\n",
            "Requirement already satisfied: gym[atari,classic_control]>=0.10.9 in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]==2.8.0) (0.17.3)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]==2.8.0) (1.19.5)\n",
            "Requirement already satisfied: cloudpickle>=0.5.5 in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]==2.8.0) (1.3.0)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]==2.8.0) (1.1.5)\n",
            "Collecting mpi4py\n",
            "  Downloading mpi4py-3.1.3.tar.gz (2.5 MB)\n",
            "\u001b[K     |████████████████████████████████| 2.5 MB 53.5 MB/s \n",
            "\u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "    Preparing wheel metadata ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from gym[atari,classic_control]>=0.10.9->stable-baselines[mpi]==2.8.0) (1.5.0)\n",
            "Requirement already satisfied: atari-py~=0.2.0 in /usr/local/lib/python3.7/dist-packages (from gym[atari,classic_control]>=0.10.9->stable-baselines[mpi]==2.8.0) (0.2.9)\n",
            "Requirement already satisfied: Pillow in /usr/local/lib/python3.7/dist-packages (from gym[atari,classic_control]>=0.10.9->stable-baselines[mpi]==2.8.0) (7.1.2)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from atari-py~=0.2.0->gym[atari,classic_control]>=0.10.9->stable-baselines[mpi]==2.8.0) (1.15.0)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from pyglet<=1.5.0,>=1.4.0->gym[atari,classic_control]>=0.10.9->stable-baselines[mpi]==2.8.0) (0.16.0)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->stable-baselines[mpi]==2.8.0) (3.0.6)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->stable-baselines[mpi]==2.8.0) (2.8.2)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->stable-baselines[mpi]==2.8.0) (0.11.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->stable-baselines[mpi]==2.8.0) (1.3.2)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->stable-baselines[mpi]==2.8.0) (2018.9)\n",
            "Building wheels for collected packages: mpi4py\n",
            "  Building wheel for mpi4py (PEP 517) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for mpi4py: filename=mpi4py-3.1.3-cp37-cp37m-linux_x86_64.whl size=2185303 sha256=2ddd08c46b5a1eca4816159e7b398edc426ae59fe8e8098477de2e87042d4916\n",
            "  Stored in directory: /root/.cache/pip/wheels/7a/07/14/6a0c63fa2c6e473c6edc40985b7d89f05c61ff25ee7f0ad9ac\n",
            "Successfully built mpi4py\n",
            "Installing collected packages: stable-baselines, mpi4py\n",
            "Successfully installed mpi4py-3.1.3 stable-baselines-2.8.0\n"
          ]
        },
        {
          "data": {
            "text/plain": []
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%shell\n",
        "\n",
        "sudo apt-get update && sudo apt-get install cmake libopenmpi-dev zlib1g-dev\n",
        "\n",
        "pip install stable-baselines[mpi]==2.8.0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FDfhzUyX-XY9"
      },
      "source": [
        "- For Windows: \n",
        "    - First install [MPI for Windows](https://www.microsoft.com/en-us/download/details.aspx?id=57467) (you need to download and install `msmpisetup.exe`)\n",
        "    - Then run this command in Prompt: `pip install stable-baselines[mpi]==2.8.0`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0-VlhRXBP-6w"
      },
      "source": [
        "install ROMs which needed for creating atari env"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ecaZyKqFPdyt",
        "outputId": "7abaa8b1-4491-49de-dce5-90f9233c6d93"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting unrar\n",
            "  Downloading unrar-0.4-py3-none-any.whl (25 kB)\n",
            "Installing collected packages: unrar\n",
            "Successfully installed unrar-0.4\n",
            "\n",
            "UNRAR 5.50 freeware      Copyright (c) 1993-2017 Alexander Roshal\n",
            "\n",
            "\n",
            "Extracting from Roms.rar\n",
            "\n",
            "Extracting  HC ROMS.zip                                                  \b\b\b\b 36%\b\b\b\b\b  OK \n",
            "Extracting  ROMS.zip                                                     \b\b\b\b 74%\b\b\b\b 99%\b\b\b\b\b  OK \n",
            "All OK\n",
            "copying adventure.bin from ROMS/Adventure (1980) (Atari, Warren Robinett) (CX2613, CX2613P) (PAL).bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/adventure.bin\n",
            "copying air_raid.bin from ROMS/Air Raid (Men-A-Vision) (PAL) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/air_raid.bin\n",
            "copying alien.bin from ROMS/Alien (1982) (20th Century Fox Video Games, Douglas 'Dallas North' Neubauer) (11006) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/alien.bin\n",
            "copying amidar.bin from ROMS/Amidar (1982) (Parker Brothers, Ed Temple) (PB5310) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/amidar.bin\n",
            "copying assault.bin from ROMS/Assault (AKA Sky Alien) (1983) (Bomb - Onbase) (CA281).bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/assault.bin\n",
            "copying asterix.bin from ROMS/Asterix (AKA Taz) (07-27-1983) (Atari, Jerome Domurat, Steve Woita) (CX2696) (Prototype).bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/asterix.bin\n",
            "copying asteroids.bin from ROMS/Asteroids (1981) (Atari, Brad Stewart - Sears) (CX2649 - 49-75163) [no copyright] ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/asteroids.bin\n",
            "copying atlantis.bin from ROMS/Atlantis (Lost City of Atlantis) (1982) (Imagic, Dennis Koble) (720103-1A, 720103-1B, IA3203, IX-010-04) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/atlantis.bin\n",
            "copying bank_heist.bin from ROMS/Bank Heist (Bonnie & Clyde, Cops 'n' Robbers, Hold-Up, Roaring 20's) (1983) (20th Century Fox Video Games, Bill Aspromonte) (11012) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/bank_heist.bin\n",
            "copying battle_zone.bin from ROMS/Battlezone (1983) (Atari - GCC, Mike Feinstein, Brad Rice) (CX2681) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/battle_zone.bin\n",
            "copying beam_rider.bin from ROMS/Beamrider (1984) (Activision - Cheshire Engineering, David Rolfe, Larry Zwick) (AZ-037-04) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/beam_rider.bin\n",
            "copying berzerk.bin from ROMS/Berzerk (1982) (Atari, Dan Hitchens - Sears) (CX2650 - 49-75168) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/berzerk.bin\n",
            "copying bowling.bin from ROMS/Bowling (1979) (Atari, Larry Kaplan - Sears) (CX2628 - 6-99842, 49-75117) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/bowling.bin\n",
            "copying boxing.bin from ROMS/Boxing - La Boxe (1980) (Activision, Bob Whitehead) (AG-002, CAG-002, AG-002-04) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/boxing.bin\n",
            "copying breakout.bin from ROMS/Breakout - Breakaway IV (Paddle) (1978) (Atari, Brad Stewart - Sears) (CX2622 - 6-99813, 49-75107) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/breakout.bin\n",
            "copying carnival.bin from ROMS/Carnival (1982) (Coleco - Woodside Design Associates, Steve 'Jessica Stevens' Kitchen) (2468) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/carnival.bin\n",
            "copying centipede.bin from ROMS/Centipede (1983) (Atari - GCC) (CX2676) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/centipede.bin\n",
            "copying chopper_command.bin from ROMS/Chopper Command (1982) (Activision, Bob Whitehead) (AX-015, AX-015-04) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/chopper_command.bin\n",
            "copying crazy_climber.bin from ROMS/Crazy Climber (1983) (Atari - Roklan, Joe Gaucher, Alex Leavens) (CX2683) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/crazy_climber.bin\n",
            "copying defender.bin from ROMS/Defender (1982) (Atari, Robert C. Polaro, Alan J. Murphy - Sears) (CX2609 - 49-75186) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/defender.bin\n",
            "copying demon_attack.bin from ROMS/Demon Attack (Death from Above) (1982) (Imagic, Rob Fulop) (720000-200, 720101-1B, 720101-1C, IA3200, IA3200C, IX-006-04) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/demon_attack.bin\n",
            "copying donkey_kong.bin from ROMS/Donkey Kong (1982) (Coleco - Woodside Design Associates - Imaginative Systems Software, Garry Kitchen) (2451) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/donkey_kong.bin\n",
            "copying double_dunk.bin from ROMS/Double Dunk (Super Basketball) (1989) (Atari, Matthew L. Hubbard) (CX26159) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/double_dunk.bin\n",
            "copying elevator_action.bin from ROMS/Elevator Action (1983) (Atari, Dan Hitchens) (CX26126) (Prototype) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/elevator_action.bin\n",
            "copying enduro.bin from ROMS/Enduro (1983) (Activision, Larry Miller) (AX-026, AX-026-04) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/enduro.bin\n",
            "copying fishing_derby.bin from ROMS/Fishing Derby (1980) (Activision, David Crane) (AG-004) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/fishing_derby.bin\n",
            "copying freeway.bin from ROMS/Freeway (1981) (Activision, David Crane) (AG-009, AG-009-04) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/freeway.bin\n",
            "copying frogger.bin from ROMS/Frogger (1982) (Parker Brothers, Ed English, David Lamkins) (PB5300) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/frogger.bin\n",
            "copying frostbite.bin from ROMS/Frostbite (1983) (Activision, Steve Cartwright) (AX-031) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/frostbite.bin\n",
            "copying galaxian.bin from ROMS/Galaxian (1983) (Atari - GCC, Mark Ackerman, Tom Calderwood, Glenn Parker) (CX2684) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/galaxian.bin\n",
            "copying gopher.bin from ROMS/Gopher (Gopher Attack) (1982) (U.S. Games Corporation - JWDA, Sylvia Day, Todd Marshall, Robin McDaniel, Henry Will IV) (VC2001) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/gopher.bin\n",
            "copying gravitar.bin from ROMS/Gravitar (1983) (Atari, Dan Hitchens, Mimi Nyden) (CX2685) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/gravitar.bin\n",
            "copying hero.bin from ROMS/H.E.R.O. (1984) (Activision, John Van Ryzin) (AZ-036-04) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/hero.bin\n",
            "copying ice_hockey.bin from ROMS/Ice Hockey - Le Hockey Sur Glace (1981) (Activision, Alan Miller) (AX-012, CAX-012, AX-012-04) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/ice_hockey.bin\n",
            "copying jamesbond.bin from ROMS/James Bond 007 (James Bond Agent 007) (1984) (Parker Brothers - On-Time Software, Joe Gaucher, Louis Marbel) (PB5110) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/jamesbond.bin\n",
            "copying journey_escape.bin from ROMS/Journey Escape (1983) (Data Age, J. Ray Dettling) (112-006) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/journey_escape.bin\n",
            "copying kaboom.bin from ROMS/Kaboom! (Paddle) (1981) (Activision, Larry Kaplan, David Crane) (AG-010, AG-010-04) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/kaboom.bin\n",
            "copying kangaroo.bin from ROMS/Kangaroo (1983) (Atari - GCC, Kevin Osborn) (CX2689) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/kangaroo.bin\n",
            "copying keystone_kapers.bin from ROMS/Keystone Kapers - Raueber und Gendarm (1983) (Activision, Garry Kitchen - Ariola) (EAX-025, EAX-025-04I - 711 025-725) (PAL).bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/keystone_kapers.bin\n",
            "copying king_kong.bin from ROMS/King Kong (1982) (Tigervision - Software Electronics Corporation, Karl T. Olinger - Teldec) (7-001 - 3.60001 VE) (PAL).bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/king_kong.bin\n",
            "copying koolaid.bin from ROMS/Kool-Aid Man (Kool Aid Pitcher Man) (1983) (M Network, Stephen Tatsumi, Jane Terjung - Kool Aid) (MT4648) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/koolaid.bin\n",
            "copying krull.bin from ROMS/Krull (1983) (Atari, Jerome Domurat, Dave Staugas) (CX2682) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/krull.bin\n",
            "copying kung_fu_master.bin from ROMS/Kung-Fu Master (1987) (Activision - Imagineering, Dan Kitchen, Garry Kitchen) (AG-039-04) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/kung_fu_master.bin\n",
            "copying laser_gates.bin from ROMS/Laser Gates (AKA Innerspace) (1983) (Imagic, Dan Oliver) (720118-2A, 13208, EIX-007-04I) (PAL).bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/laser_gates.bin\n",
            "copying lost_luggage.bin from ROMS/Lost Luggage (Airport Mayhem) (1982) (Apollo - Games by Apollo, Larry Minor, Ernie Runyon, Ed Salvo) (AP-2004) [no opening scene] ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/lost_luggage.bin\n",
            "copying montezuma_revenge.bin from ROMS/Montezuma's Revenge - Featuring Panama Joe (1984) (Parker Brothers - JWDA, Henry Will IV) (PB5760) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/montezuma_revenge.bin\n",
            "copying mr_do.bin from ROMS/Mr. Do! (1983) (CBS Electronics, Ed English) (4L4478) (PAL).bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/mr_do.bin\n",
            "copying ms_pacman.bin from ROMS/Ms. Pac-Man (1983) (Atari - GCC, Mark Ackerman, Glenn Parker) (CX2675) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/ms_pacman.bin\n",
            "copying name_this_game.bin from ROMS/Name This Game (Guardians of Treasure) (1983) (U.S. Games Corporation - JWDA, Roger Booth, Sylvia Day, Ron Dubren, Todd Marshall, Robin McDaniel, Wes Trager, Henry Will IV) (VC1007) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/name_this_game.bin\n",
            "copying pacman.bin from ROMS/Pac-Man (1982) (Atari, Tod Frye) (CX2646) (PAL).bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/pacman.bin\n",
            "copying phoenix.bin from ROMS/Phoenix (1983) (Atari - GCC, Mike Feinstein, John Mracek) (CX2673) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/phoenix.bin\n",
            "copying video_pinball.bin from ROMS/Pinball (AKA Video Pinball) (Zellers).bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/video_pinball.bin\n",
            "copying pitfall.bin from ROMS/Pitfall! - Pitfall Harry's Jungle Adventure (Jungle Runner) (1982) (Activision, David Crane) (AX-018, AX-018-04) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/pitfall.bin\n",
            "copying pooyan.bin from ROMS/Pooyan (1983) (Konami) (RC 100-X 02) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/pooyan.bin\n",
            "copying private_eye.bin from ROMS/Private Eye (1984) (Activision, Bob Whitehead) (AG-034-04) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/private_eye.bin\n",
            "copying qbert.bin from ROMS/Q-bert (1983) (Parker Brothers - Western Technologies, Dave Hampton, Tom Sloper) (PB5360) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/qbert.bin\n",
            "copying riverraid.bin from ROMS/River Raid (1982) (Activision, Carol Shaw) (AX-020, AX-020-04) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/riverraid.bin\n",
            "copying road_runner.bin from patched version of ROMS/Road Runner (1989) (Atari - Bobco, Robert C. Polaro) (CX2663) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/road_runner.bin\n",
            "copying robotank.bin from ROMS/Robot Tank (Robotank) (1983) (Activision, Alan Miller) (AZ-028, AG-028-04) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/robotank.bin\n",
            "copying seaquest.bin from ROMS/Seaquest (1983) (Activision, Steve Cartwright) (AX-022) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/seaquest.bin\n",
            "copying sir_lancelot.bin from ROMS/Sir Lancelot (1983) (Xonox - K-Tel Software - Product Guild, Anthony R. Henderson) (99006, 6220) (PAL).bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/sir_lancelot.bin\n",
            "copying skiing.bin from ROMS/Skiing - Le Ski (1980) (Activision, Bob Whitehead) (AG-005, CAG-005, AG-005-04) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/skiing.bin\n",
            "copying solaris.bin from ROMS/Solaris (The Last Starfighter, Star Raiders II, Universe) (1986) (Atari, Douglas Neubauer, Mimi Nyden) (CX26136) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/solaris.bin\n",
            "copying space_invaders.bin from ROMS/Space Invaders (1980) (Atari, Richard Maurer - Sears) (CX2632 - 49-75153) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/space_invaders.bin\n",
            "copying star_gunner.bin from ROMS/Stargunner (1983) (Telesys, Alex Leavens) (1005) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/star_gunner.bin\n",
            "copying surround.bin from ROMS/Surround (32 in 1) (Bit Corporation) (R320).bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/surround.bin\n",
            "copying tennis.bin from ROMS/Tennis - Le Tennis (1981) (Activision, Alan Miller) (AG-007, CAG-007) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/tennis.bin\n",
            "copying time_pilot.bin from ROMS/Time Pilot (1983) (Coleco - Woodside Design Associates, Harley H. Puthuff Jr.) (2663) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/time_pilot.bin\n",
            "copying trondead.bin from ROMS/TRON - Deadly Discs (TRON Joystick) (1983) (M Network - INTV - APh Technological Consulting, Jeff Ronne, Brett Stutz) (MT5662) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/trondead.bin\n",
            "copying tutankham.bin from ROMS/Tutankham (1983) (Parker Brothers, Dave Engman, Dawn Stockbridge) (PB5340) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/tutankham.bin\n",
            "copying up_n_down.bin from ROMS/Up 'n Down (1984) (SEGA - Beck-Tech, Steve Beck, Phat Ho) (009-01) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/up_n_down.bin\n",
            "copying venture.bin from ROMS/Venture (1982) (Coleco, Joseph Biel) (2457) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/venture.bin\n",
            "copying pong.bin from ROMS/Video Olympics - Pong Sports (Paddle) (1977) (Atari, Joe Decuir - Sears) (CX2621 - 99806, 6-99806, 49-75104) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/pong.bin\n",
            "copying wizard_of_wor.bin from ROMS/Wizard of Wor (1982) (CBS Electronics - Roklan, Joe Hellesen, Joe Wagner) (M8774, M8794) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/wizard_of_wor.bin\n",
            "copying yars_revenge.bin from ROMS/Yars' Revenge (Time Freeze) (1982) (Atari, Howard Scott Warshaw - Sears) (CX2655 - 49-75167) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/yars_revenge.bin\n",
            "copying zaxxon.bin from ROMS/Zaxxon (1983) (Coleco) (2454) ~.bin to /usr/local/lib/python3.7/dist-packages/atari_py/atari_roms/zaxxon.bin\n"
          ]
        }
      ],
      "source": [
        "import urllib.request\n",
        "urllib.request.urlretrieve('http://www.atarimania.com/roms/Roms.rar','Roms.rar')\n",
        "!pip install unrar\n",
        "!unrar x Roms.rar\n",
        "!mkdir rars\n",
        "!mv HC\\ ROMS.zip   rars\n",
        "!mv ROMS.zip  rars\n",
        "!python -m atari_py.import_roms rars"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A1nxP4EN-aXC"
      },
      "source": [
        "### 2. Import Libraries:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Eb7L5Ty8-a-6",
        "outputId": "f395914b-6790-4336-c2e2-f7ad6c29272d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "TensorFlow 1.x selected.\n",
            "WARNING:tensorflow:\n",
            "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "  * https://github.com/tensorflow/io (for I/O related ops)\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n"
          ]
        }
      ],
      "source": [
        "import random, os.path, math, glob, csv, os\n",
        "import numpy as np\n",
        "\n",
        "from timeit import default_timer as timer\n",
        "from datetime import timedelta\n",
        "\n",
        "import matplotlib\n",
        "%matplotlib inline\n",
        "from IPython.display import clear_output\n",
        "from plot import plot_all_data \n",
        "\n",
        "import torch\n",
        "import torch.optim as optim\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "\n",
        "# ######################## #\n",
        "# USE ONLY IN GOOGLE COLAB #\n",
        "%tensorflow_version 1.x \n",
        "# ######################## #\n",
        "\n",
        "import gym\n",
        "from gym.spaces.box import Box\n",
        "from stable_baselines import bench\n",
        "from stable_baselines.common.atari_wrappers import make_atari, wrap_deepmind"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UppXT1Jw_htY"
      },
      "source": [
        "### 2. Hyperparameters"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "483i_bGfEhXD"
      },
      "outputs": [],
      "source": [
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "\n",
        "# Epsilon variables for epsilon-greedy:\n",
        "epsilon_start    = 1.0\n",
        "epsilon_final    = 0.01\n",
        "epsilon_decay    = 30000\n",
        "epsilon_by_frame = lambda frame_idx: epsilon_final + (epsilon_start - epsilon_final) * math.exp(-1. * frame_idx / epsilon_decay) \n",
        "\n",
        "# Misc agent variables\n",
        "GAMMA = 0.99 \n",
        "LR    = 1e-4 \n",
        "\n",
        "# Memory\n",
        "TARGET_NET_UPDATE_FREQ = 1000 \n",
        "EXP_REPLAY_SIZE        = 100000 \n",
        "BATCH_SIZE             = 32 \n",
        "\n",
        "# Learning control variables\n",
        "LEARN_START = 10000\n",
        "MAX_FRAMES  = 200000 # Probably takes about an hour training. You can increase it if you have time!\n",
        "UPDATE_FREQ = 1 \n",
        "\n",
        "# Data logging parameters\n",
        "ACTION_SELECTION_COUNT_FREQUENCY = 1000 \n",
        "\n",
        "#DRQN Parameters\n",
        "SEQUENCE_LENGTH = 8"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vVyNn9yDRt2X"
      },
      "source": [
        "### 3. Wrapper"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5z4G9Xg0_ewj"
      },
      "outputs": [],
      "source": [
        "class WrapPOMDP(gym.ObservationWrapper):\n",
        "    def __init__(self, env=None):\n",
        "        super(WrapPOMDP, self).__init__(env)\n",
        "        obs_shape = self.observation_space.shape\n",
        "        self.observation_space = Box(\n",
        "            self.observation_space.low[0, 0, 0],\n",
        "            self.observation_space.high[0, 0, 0],\n",
        "            [obs_shape[2], obs_shape[1], obs_shape[0]],\n",
        "            dtype=self.observation_space.dtype)\n",
        "\n",
        "    def observation(self, observation):\n",
        "        # this method change MDP into POMDP\n",
        "        pomdp = np.random.uniform()\n",
        "        if pomdp >= 0.5:\n",
        "          return observation.transpose(2, 0, 1)\n",
        "        else:\n",
        "          return observation.transpose(2, 0, 1) * 0.0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vHjBFn-qR0jB"
      },
      "source": [
        "### 4. Abstract Agent Class (3 Points)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "amZqXtiGSEEw"
      },
      "outputs": [],
      "source": [
        "class BaseAgent():\n",
        "    def __init__(self, model, target_model, log_dir, env):\n",
        "        self.device = device\n",
        "        self.gamma = GAMMA\n",
        "        self.lr = LR\n",
        "        self.target_net_update_freq = TARGET_NET_UPDATE_FREQ\n",
        "        self.experience_replay_size = EXP_REPLAY_SIZE\n",
        "        self.batch_size = BATCH_SIZE\n",
        "        self.learn_start = LEARN_START\n",
        "        self.update_freq = UPDATE_FREQ\n",
        "        self.update_count = 0\n",
        "        self.nstep_buffer = []\n",
        "        self.rewards = []\n",
        "        self.model = model\n",
        "        self.target_model = target_model\n",
        "        #################################################################################\n",
        "        #                          COMPLETE THE FOLLOWING SECTION                       #\n",
        "        #################################################################################\n",
        "        # compelete this method\n",
        "        # * save log_dir\n",
        "        # * save env\n",
        "        # * make a list of action selections\n",
        "        # * using load_state_dict, share learnable parameters (i.e. weights and biases) of\n",
        "        #   self.model with self.target_model \n",
        "        # * move both model to correct device\n",
        "        # * use Adam optimizer\n",
        "        # * set both model to train mode\n",
        "        #################################################################################\n",
        "        self.log_dir = None\n",
        "        self.env = None\n",
        "        self.action_selections = None\n",
        "        self.target_model.load_state_dict(self.model.state_dict())\n",
        "        self.optimizer = None\n",
        "        #################################################################################\n",
        "        #                                   THE END                                     #\n",
        "        #################################################################################\n",
        "\n",
        "    def update_target_model(self):\n",
        "        # update target model:\n",
        "        self.update_count += 1\n",
        "        self.update_count = self.update_count % self.target_net_update_freq\n",
        "        if self.update_count == 0:\n",
        "            self.target_model.load_state_dict(self.model.state_dict())\n",
        "\n",
        "    def save_sigma_param_magnitudes(self, tstep):\n",
        "        with torch.no_grad():\n",
        "            sum_, count = 0.0, 0.0\n",
        "            for name, param in self.model.named_parameters():\n",
        "                if param.requires_grad and 'sigma' in name:\n",
        "                    sum_ += torch.sum(param.abs()).item()\n",
        "                    count += np.prod(param.shape)\n",
        "            if count > 0:\n",
        "                with open(os.path.join(self.log_dir, 'sig_param_mag.csv'), 'a') as f:\n",
        "                    writer = csv.writer(f)\n",
        "                    writer.writerow((tstep, sum_ / count))\n",
        "\n",
        "    def save_td(self, td, tstep):\n",
        "        with open(os.path.join(self.log_dir, 'td.csv'), 'a') as f:\n",
        "            writer = csv.writer(f)\n",
        "            writer.writerow((tstep, td))\n",
        "\n",
        "    def save_reward(self, reward):\n",
        "        self.rewards.append(reward)\n",
        "\n",
        "    def save_action(self, action, tstep):\n",
        "        self.action_selections[int(action)] += 1.0 / self.action_log_frequency\n",
        "        if (tstep + 1) % self.action_log_frequency == 0:\n",
        "            with open(os.path.join(self.log_dir, 'action_log.csv'), 'a') as f:\n",
        "                writer = csv.writer(f)\n",
        "                writer.writerow(list([tstep] + self.action_selections))\n",
        "            self.action_selections = [0 for _ in range(len(self.action_selections))]\n",
        "\n",
        "    def get_max_next_state_action(self, next_states):\n",
        "        return self.target_model(next_states).max(dim=1)[1].view(-1, 1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xSC67O8y-rVe"
      },
      "source": [
        "# 1. Simple DQN"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rBKeONfcat2R"
      },
      "source": [
        "We will implement a DQN model with experience replay. We implement a class for experience replay  `ExperienceReplayMemory`, for extarct features of observed picture of game we implement a class `DQN` that uses a CNN and inheritance from `nn.Module`. We implemented a wrapper class `WrapPyTorch` that you will use it in training loop."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ft7jEkrq-_In"
      },
      "source": [
        "## 1.1. Experience Replay (3 Points)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "--3niP_1_B0G"
      },
      "outputs": [],
      "source": [
        "#################################################################################\n",
        "#                          COMPLETE THE FOLLOWING SECTION                       #\n",
        "#################################################################################\n",
        "# complete push and sample methods.\n",
        "#################################################################################\n",
        "class ExperienceReplayMemory:\n",
        "    def __init__(self, capacity):\n",
        "        self.capacity = capacity\n",
        "        self.memory = []\n",
        "\n",
        "    def push(self, transition):\n",
        "        pass\n",
        "\n",
        "    def sample(self, batch_size):\n",
        "        return None\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.memory)\n",
        "#################################################################################\n",
        "#                                   THE END                                     #\n",
        "#################################################################################  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x02ei6IA_D7B"
      },
      "source": [
        "## 1.2. Network Declaration (4 Points)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ImVKJNl6_EaO"
      },
      "outputs": [],
      "source": [
        "class DQN(nn.Module):\n",
        "    def __init__(self, input_shape, num_actions):\n",
        "        super(DQN, self).__init__()\n",
        "        \n",
        "        self.input_shape = input_shape\n",
        "        self.num_actions = num_actions\n",
        "        #################################################################################\n",
        "        #                          COMPLETE THE FOLLOWING SECTION                       #\n",
        "        #################################################################################\n",
        "        # Initialize CNN Model :\n",
        "        # conv1: out_channels:32, kernel_size=8, stride=4\n",
        "        # conv2: out_channels:64, kernel_size=4, stride=2\n",
        "        # conv3: out_channels:64, kernel_size=3, stride=1\n",
        "        # fc1(512)\n",
        "        # fc2(512)\n",
        "        #################################################################################\n",
        "        pass\n",
        "        #################################################################################\n",
        "        #                                   THE END                                     #\n",
        "        #################################################################################\n",
        "\n",
        "    def forward(self, x):\n",
        "        #################################################################################\n",
        "        #                          COMPLETE THE FOLLOWING SECTION                       #\n",
        "        #################################################################################\n",
        "        # complete forward pass using initialized CNN Model. use Relu activation function \n",
        "        # for conv1, conv2, conv3, and fc1.  \n",
        "        #################################################################################\n",
        "        pass\n",
        "\n",
        "        return x\n",
        "        #################################################################################\n",
        "        #                                   THE END                                     #\n",
        "        #################################################################################   \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XZV6bdgM_Lcl"
      },
      "source": [
        " ## 1.3. Agent (6 Points)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IEgTi4Gd_MwE"
      },
      "outputs": [],
      "source": [
        "class Model():\n",
        "    def __init__(self, env=None, log_dir):\n",
        "        self.device = device\n",
        "\n",
        "        self.gamma = GAMMA \n",
        "        self.lr = LR\n",
        "        self.target_net_update_freq = TARGET_NET_UPDATE_FREQ\n",
        "        self.experience_replay_size = EXP_REPLAY_SIZE\n",
        "        self.batch_size = BATCH_SIZE\n",
        "        self.learn_start = LEARN_START\n",
        "        self.update_freq = UPDATE_FREQ\n",
        "        self.log_dir = log_dir\n",
        "        self.rewards = []\n",
        "        self.action_log_frequency = ACTION_SELECTION_COUNT_FREQUENCY\n",
        "        #################################################################################\n",
        "        #                          COMPLETE THE FOLLOWING SECTION                       #\n",
        "        #################################################################################\n",
        "        # do followings line by line:\n",
        "        # * make a list of action selections\n",
        "        # * use shape of observation space to save number of features\n",
        "        # * save naumber of actions\n",
        "        # * use DQN class to declare model and target model (the game is 2-player game, so\n",
        "        #   we declare 2 model)\n",
        "        # * using load_state_dict, share learnable parameters (i.e. weights and biases) of\n",
        "        #   self.model with self.target_model \n",
        "        # * use Adam optimizer\n",
        "        #################################################################################\n",
        "        self.num_feats = None\n",
        "        self.num_actions = None\n",
        "        self.model = None\n",
        "        self.target_model = None\n",
        "        super(Model, self).__init__()\n",
        "        #################################################################################\n",
        "        #                                   THE END                                     #\n",
        "        #################################################################################\n",
        "        \n",
        "        self.memory = ExperienceReplayMemory(self.experience_replay_size)\n",
        "        \n",
        "\n",
        "    def prep_minibatch(self):\n",
        "      \n",
        "        #################################################################################\n",
        "        #                          COMPLETE THE FOLLOWING SECTION                       #\n",
        "        #################################################################################\n",
        "        # random transition batch is taken from experience replay memory\n",
        "        # do followings line by line:\n",
        "        # * sample from self.memory with batch size, and save result in transitions\n",
        "        # * use transitions to save batch_state, batch_action, batch_reward, \n",
        "        #   batch_next_state as tensors\n",
        "        # * save non_final_mask,  non_final_next_states as tensors, note that sometimes \n",
        "        #   all next states are false\n",
        "        #################################################################################\n",
        "        transitions = None\n",
        "        \n",
        "        batch_state, batch_action, batch_reward, batch_next_state = None\n",
        "\n",
        "  \n",
        "\n",
        "        batch_state = None\n",
        "        batch_action = None\n",
        "        batch_reward = None\n",
        "        \n",
        "        non_final_mask = None\n",
        "\n",
        "        try: \n",
        "            non_final_next_states = None\n",
        "            empty_next_state_values = False\n",
        "        except:\n",
        "            non_final_next_states = None\n",
        "            empty_next_state_values = True\n",
        "        #################################################################################\n",
        "        #                                   THE END                                     #\n",
        "        #################################################################################\n",
        "        return batch_state, batch_action, batch_reward, non_final_next_states, non_final_mask, empty_next_state_values\n",
        "\n",
        "    def compute_loss(self, batch_vars):\n",
        "        batch_state, batch_action, batch_reward, non_final_next_states, non_final_mask, empty_next_state_values = batch_vars\n",
        "\n",
        "        #################################################################################\n",
        "        #                          COMPLETE THE FOLLOWING SECTION                       #\n",
        "        #################################################################################\n",
        "        # implement calculation of loss (you should use \"with torch.no_grad():\" for target_model)\n",
        "        #################################################################################\n",
        "\n",
        "        loss = None\n",
        "        #################################################################################\n",
        "        #                                   THE END                                     #\n",
        "        #################################################################################\n",
        "        return loss\n",
        "\n",
        "    def update(self, s, a, r, s_, frame=0):\n",
        "        #################################################################################\n",
        "        #                          COMPLETE THE FOLLOWING SECTION                       #\n",
        "        #################################################################################\n",
        "        # implement update method to optimize model \n",
        "        # * push state, action, reward and new state to memoty\n",
        "        # * note that if frame is lower than self.learn_start or frame % self.update_freq != 0\n",
        "        #   return None\n",
        "        # * take a random transition batch and compute loss\n",
        "        # * optimize the model\n",
        "        #################################################################################\n",
        "        pass\n",
        "        #################################################################################\n",
        "        #                                   THE END                                     #\n",
        "        #################################################################################\n",
        "\n",
        "        self.update_target_model()       \n",
        "        self.save_td(loss.item(), frame)\n",
        "        self.save_sigma_param_magnitudes(frame)\n",
        "\n",
        "    def get_action(self, s, eps=0.1):\n",
        "\n",
        "        #################################################################################\n",
        "        #                          COMPLETE THE FOLLOWING SECTION                       #\n",
        "        #################################################################################  \n",
        "         # implement get_action method (epsilon greedy)\n",
        "         # you should use \"with torch.no_grad():\"\n",
        "         ################################################################################# \n",
        "        pass\n",
        "\n",
        "        #################################################################################\n",
        "        #                                   THE END                                     #\n",
        "        #################################################################################\n",
        "\n",
        "    def update_target_model(self):\n",
        "      # update target model:\n",
        "        self.update_count+=1\n",
        "        self.update_count = self.update_count % self.target_net_update_freq\n",
        "        if self.update_count == 0:\n",
        "            self.target_model.load_state_dict(self.model.state_dict())\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nOJD_Sc2_Ow2"
      },
      "source": [
        " ## 1.3. Training Loop (4 Points)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1oOHYnO1_Pzp"
      },
      "outputs": [],
      "source": [
        "start=timer()\n",
        "\n",
        "log_dir = \"/tmp/gym/dqn\"\n",
        "try:\n",
        "    os.makedirs(log_dir)\n",
        "except OSError:\n",
        "    files = glob.glob(os.path.join(log_dir, '*.monitor.csv')) \\\n",
        "        + glob.glob(os.path.join(log_dir, '*td.csv')) \\\n",
        "        + glob.glob(os.path.join(log_dir, '*sig_param_mag.csv')) \\\n",
        "        + glob.glob(os.path.join(log_dir, '*action_log.csv'))\n",
        "    for f in files:\n",
        "        os.remove(f)\n",
        "\n",
        "#################################################################################\n",
        "#                          COMPLETE THE FOLLOWING SECTION                       #\n",
        "#################################################################################\n",
        "# do followings line by line:\n",
        "# * use \"make_atari\" wrapper and save \"PongNoFrameskip-v4\" game to env\n",
        "# * use \"bench.Monitor\" wrapper to know the episode reward, length, time and other data.\n",
        "# * use \"wrap_deepmind\" wrapper to configure environment for DeepMind-style Atari.\n",
        "# * use *WrapPyTorch*\n",
        "# * save model\n",
        "# * implement training loop\n",
        "#################################################################################\n",
        "env_id = \"PongNoFrameskip-v4\"\n",
        "env =  None\n",
        "\n",
        "model  = None\n",
        "\n",
        "episode_reward = 0\n",
        "observation = env.reset()\n",
        "for frame_idx in range(1, MAX_FRAMES + 1):\n",
        "    epsilon = None\n",
        "\n",
        "    action = None\n",
        "\n",
        "    model.save_action(action, frame_idx) #log action selection\n",
        "\n",
        "    prev_observation = None\n",
        "    observation, reward, done, _ = None\n",
        "\n",
        "    ....\n",
        "\n",
        "    if done:\n",
        "        pass\n",
        "        \n",
        "#################################################################################\n",
        "#                                   THE END                                     #\n",
        "################################################################################# \n",
        "    if frame_idx % 1000 == 0:\n",
        "        try:\n",
        "            clear_output(True)\n",
        "            plot_all_data(log_dir, env_id, 'DQN', MAX_FRAMES, bin_size=(10, 100, 100, 1), smooth=1, time=timedelta(seconds=int(timer()-start)), ipynb=True)\n",
        "        except IOError:\n",
        "            pass\n",
        " \n",
        "env.close()\n",
        "plot_all_data(log_dir, env_id, 'DQN', MAX_FRAMES, bin_size=(10, 100, 100, 1), smooth=1, time=timedelta(seconds=int(timer()-start)), ipynb=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eiqt6Ssb-utm"
      },
      "source": [
        "# 2. DRQN"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XoECt8BQCseR"
      },
      "source": [
        "## 2.1. Experience Replay (4 Points)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IcjMhORZCseR"
      },
      "outputs": [],
      "source": [
        "class RecurrentExperienceReplayMemory:\n",
        "    def __init__(self, capacity, sequence_length=10):\n",
        "        self.capacity = capacity\n",
        "        self.memory = []\n",
        "        self.seq_length = sequence_length\n",
        "\n",
        "    def push(self, transition):\n",
        "        #################################################################################\n",
        "        #                          COMPLETE THE FOLLOWING SECTION                       #\n",
        "        #################################################################################\n",
        "        # complete push method.\n",
        "        #################################################################################\n",
        "        pass\n",
        "        #################################################################################\n",
        "        #                                   THE END                                     #\n",
        "        #################################################################################  \n",
        "\n",
        "    def sample(self, batch_size):\n",
        "        #################################################################################\n",
        "        #                          COMPLETE THE FOLLOWING SECTION                       #\n",
        "        #################################################################################\n",
        "        # complete sample method.\n",
        "        # notice that you should take these tips into consideration\n",
        "        # * sample here will be trajectory not transition\n",
        "        # * should use padding if trajectories aren't in same len\n",
        "        #################################################################################\n",
        "        samples = []\n",
        "        pass\n",
        "        #################################################################################\n",
        "        #                                   THE END                                     #\n",
        "        #################################################################################  \n",
        "        return samples\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.memory)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qZuglsfRCseS"
      },
      "source": [
        "## 2.2. Network Declaration (4 Points)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GUnIpU5-CseS"
      },
      "outputs": [],
      "source": [
        "class RecurrentDQN(nn.Module):\n",
        "    def __init__(self, input_shape, num_actions):\n",
        "        super(RecurrentDQN,self).__init__()\n",
        "        \n",
        "        self.input_shape = input_shape\n",
        "        self.num_actions = num_actions\n",
        "        #################################################################################\n",
        "        #                          COMPLETE THE FOLLOWING SECTION                       #\n",
        "        #################################################################################\n",
        "        # Initialize CNN Model :\n",
        "        # conv1: out_channels:32, kernel_size=8, stride=4\n",
        "        # conv2: out_channels:64, kernel_size=4, stride=2\n",
        "        # conv3: out_channels:64, kernel_size=3, stride=1\n",
        "        # fc2(256)\n",
        "        # GRU:  input_size: 256  hidden_size=256\n",
        "        # fc2(256)\n",
        "        #################################################################################\n",
        "        pass\n",
        "        #################################################################################\n",
        "        #                                   THE END                                     #\n",
        "        #################################################################################\n",
        "\n",
        "    def forward(self, x, bsize, time_step, hidden_state, cell_state):\n",
        "        #################################################################################\n",
        "        #                          COMPLETE THE FOLLOWING SECTION                       #\n",
        "        #################################################################################\n",
        "        # complete forward pass using initialized CNN and GRU Model. use Relu activation\n",
        "        # function for conv1, conv2, conv3 and fc1 . \n",
        "        #################################################################################\n",
        "        pass  \n",
        "        return x, hidden\n",
        "      \n",
        "      def init_hidden_states(self,bsize):\n",
        "        h = None\n",
        "        pass\n",
        "        return h\n",
        "        #################################################################################\n",
        "        #                                   THE END                                     #\n",
        "        #################################################################################   \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "12RQclOiCseS"
      },
      "source": [
        " ## 2.3. Agent (6 Points)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 130
        },
        "id": "5wat5Mt0CseS",
        "outputId": "9c53cb18-e980-45cd-bc66-558db5b937e8"
      },
      "outputs": [
        {
          "ename": "IndentationError",
          "evalue": "ignored",
          "output_type": "error",
          "traceback": [
            "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-1-865c4d6d8781>\"\u001b[0;36m, line \u001b[0;32m185\u001b[0m\n\u001b[0;31m    self.seq = [np.zeros(self.num_feats) for j in range(self.sequence_length)]\u001b[0m\n\u001b[0m       ^\u001b[0m\n\u001b[0;31mIndentationError\u001b[0m\u001b[0;31m:\u001b[0m expected an indented block\n"
          ]
        }
      ],
      "source": [
        "class RecurrentModel():\n",
        "    def __init__(self, env=None, log_dir):\n",
        "        self.sequence_length = SEQUENCE_LENGTH\n",
        "        #################################################################################\n",
        "        #                          COMPLETE THE FOLLOWING SECTION                       #\n",
        "        #################################################################################\n",
        "        # do followings line by line:\n",
        "        # * use shape of observation space to save number of features\n",
        "        # * save naumber of actions\n",
        "        # * use DRQN class to declare model and target model \n",
        "        # * call parent class constructor\n",
        "        # * declare memory\n",
        "        # * reset hidden state\n",
        "        #################################################################################\n",
        "        self.num_feats = None\n",
        "        self.num_actions = None\n",
        "        model = None\n",
        "        target_model = None\n",
        "        super(RecurrentModel, self).__init__()\n",
        "        self.memory = None\n",
        "        #################################################################################\n",
        "        #                                   THE END                                     #\n",
        "        #################################################################################\n",
        "        \n",
        "\n",
        "    def prep_minibatch(self):\n",
        "      \n",
        "        #################################################################################\n",
        "        #                          COMPLETE THE FOLLOWING SECTION                       #\n",
        "        #################################################################################\n",
        "        # random transition batch is taken from experience replay memory\n",
        "        # do followings line by line:\n",
        "        # * sample from self.memory with batch size, and save result in transitions\n",
        "        # * use transitions to save batch_state, batch_action, batch_reward, \n",
        "        #   batch_next_state as tensors\n",
        "        # * reshape batch_state, batch_action, batch_reward, batch_next_state into \n",
        "        #   (batch_size, sequence_length, feat_size)\n",
        "        # * get set of next states for end of each sequence\n",
        "        # * save non_final_mask,  non_final_next_states as tensors, note that sometimes \n",
        "        #   all next states are false\n",
        "        #################################################################################\n",
        "        try: \n",
        "            non_final_next_states = None\n",
        "            empty_next_state_values = False\n",
        "        except:\n",
        "            non_final_next_states = None\n",
        "            empty_next_state_values = True\n",
        "        #################################################################################\n",
        "        #                                   THE END                                     #\n",
        "        #################################################################################\n",
        "        return batch_state, batch_action, batch_reward, non_final_next_states, non_final_mask, empty_next_state_values\n",
        "\n",
        "    def compute_loss(self, batch_vars):\n",
        "        batch_state, batch_action, batch_reward, non_final_next_states, non_final_mask, empty_next_state_values = batch_vars\n",
        "\n",
        "        #################################################################################\n",
        "        #                          COMPLETE THE FOLLOWING SECTION                       #\n",
        "        #################################################################################\n",
        "        # implement calculation of loss (you should use \"with torch.no_grad():\" for target_model)\n",
        "        #################################################################################\n",
        "        pass\n",
        "        #################################################################################\n",
        "        #                                   THE END                                     #\n",
        "        #################################################################################\n",
        "        return loss\n",
        "\n",
        "    def update(self, s, a, r, s_, frame=0):\n",
        "        #################################################################################\n",
        "        #                          COMPLETE THE FOLLOWING SECTION                       #\n",
        "        #################################################################################\n",
        "        # implement update method to optimize model \n",
        "        # * push state, action, reward and new state to memoty\n",
        "        # * note that if frame is lower than self.learn_start or frame % self.update_freq != 0\n",
        "        #   return None\n",
        "        # * take a random transition batch and compute loss\n",
        "        # * clamp grad between -1 and 1\n",
        "        # * optimize the model \n",
        "        #################################################################################\n",
        "        pass\n",
        "        #################################################################################\n",
        "        #                                   THE END                                     #\n",
        "        #################################################################################\n",
        "\n",
        "        self.update_target_model()       \n",
        "        self.save_td(loss.item(), frame)\n",
        "        self.save_sigma_param_magnitudes(frame)\n",
        "\n",
        "    def get_action(self, s, eps=0.1):\n",
        "\n",
        "        #################################################################################\n",
        "        #                          COMPLETE THE FOLLOWING SECTION                       #\n",
        "        #################################################################################  \n",
        "         # implement get_action method (epsilon greedy)\n",
        "         # you should use \"with torch.no_grad():\"\n",
        "         ################################################################################# \n",
        "        pass\n",
        "\n",
        "        #################################################################################\n",
        "        #                                   THE END                                     #\n",
        "        #################################################################################\n",
        "\n",
        "    def reset_hx(self):\n",
        "        self.seq = [np.zeros(self.num_feats) for j in range(self.sequence_length)]\n",
        "\n",
        "    def finish_nstep(self):\n",
        "        while len(self.nstep_buffer) > 0:\n",
        "            R = sum([self.nstep_buffer[i][2] * (self.gamma ** i) for i in range(len(self.nstep_buffer))])\n",
        "            state, action, _, _ = self.nstep_buffer.pop(0)\n",
        "\n",
        "            self.memory.push((state, action, R, None))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xa9KKJA4CseT"
      },
      "source": [
        " ## 2.4. Training Loop (2 Points)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Msh8FD1kCseT"
      },
      "outputs": [],
      "source": [
        "start = timer()\n",
        "\n",
        "log_dir = \"/tmp/gym/drqn\"\n",
        "try:\n",
        "    os.makedirs(log_dir)\n",
        "except OSError:\n",
        "    files = glob.glob(os.path.join(log_dir, '*.monitor.csv')) \\\n",
        "            + glob.glob(os.path.join(log_dir, '*td.csv')) \\\n",
        "            + glob.glob(os.path.join(log_dir, '*sig_param_mag.csv')) \\\n",
        "            + glob.glob(os.path.join(log_dir, '*action_log.csv'))\n",
        "    for f in files:\n",
        "        os.remove(f)\n",
        "#################################################################################\n",
        "#                          COMPLETE THE FOLLOWING SECTION                       #\n",
        "#################################################################################\n",
        "# do followings line by line:\n",
        "# * use \"make_atari\" wrapper and save \"PongNoFrameskip-v4\" game to env\n",
        "# * use \"bench.Monitor\" wrapper to know the episode reward, length, time and other data.\n",
        "# * use \"wrap_deepmind\" wrapper to configure environment for DeepMind-style Atari.\n",
        "# * use *WrapPyTorch*\n",
        "# * save model\n",
        "# * implement training loop\n",
        "#################################################################################\n",
        "env_id = \"PongNoFrameskip-v4\"\n",
        "env =  None\n",
        "\n",
        "model  = None\n",
        "\n",
        "episode_reward = 0\n",
        "observation = env.reset()\n",
        "\n",
        "for frame_idx in range(1, MAX_FRAMES + 1):\n",
        "    epsilon = None\n",
        "\n",
        "    action = None\n",
        "\n",
        "    model.save_action(action, frame_idx) #log action selection\n",
        "\n",
        "    prev_observation = None\n",
        "    observation, reward, done, _ = None\n",
        "\n",
        "    ....\n",
        "\n",
        "    if done:\n",
        "        model.finish_nstep()\n",
        "        model.reset_hx()\n",
        "        pass\n",
        "#################################################################################\n",
        "#                                   THE END                                     #\n",
        "################################################################################# \n",
        "    if frame_idx % 10000 == 0:\n",
        "        try:\n",
        "            clear_output(True)\n",
        "            plot_all_data(log_dir, env_id, 'DRQN', MAX_FRAMES, bin_size=(10, 100, 100, 1), smooth=1,\n",
        "                          time=timedelta(seconds=int(timer() - start)), ipynb=True)\n",
        "        except IOError:\n",
        "            pass\n",
        "\n",
        "env.close()\n",
        "plot_all_data(log_dir, env_id, 'DRQN', MAX_FRAMES, bin_size=(10, 100, 100, 1), smooth=1,\n",
        "              time=timedelta(seconds=int(timer() - start)), ipynb=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BG9ftWHrhy0N"
      },
      "source": [
        " ## 2.5. Interpret Results (4 Points)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jx5uqyGEiGsz"
      },
      "source": [
        "Explain what you have seen. Is using recurrent memory improve performance? support your answer"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "DQN_DRQN.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}