009_cli_boto.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/rzl-ds/gu511/blob/master/009_cli_boto.ipynb\" target=\"_parent\">\n",
    "    <img src=\"https://colab.research.google.com/assets/colab-badge.svg\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# aws programmatic interfaces"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "up until this point, all of the work we've done with `aws` services has been via their point-and-click web console. the console is pretty good (inconsistently formatted, but generally self-explanatory), but it is not the only way to interact with `aws` services.\n",
    "\n",
    "that's good, too -- imagine you learned that the project you were about to start required you to spin up 50 `ec2` servers. think of all the fun you had clicking check boxes, scrolling through options lists, and hammering \"Next\" buttons when you started up your first server, and then multiply it times 50.\n",
    "\n",
    "a good programmer knows: anything worth doing once is worth doing *only* once. spend your time automating things, not doing them!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`aws` exposes a number of different programmatic ways to work with their services, allowing us to script menial or complicated tasks like that described above. at the current time, these tools include\n",
    "\n",
    "+ the `aws cli` (command line interface)\n",
    "    + a set of `python` scripts using the `python sdk` below to implement common `aws` service tasks in a bash or powershell script setting\n",
    "+ `SDK`s (software development kits)\n",
    "    + includes the following languages: `python`, blah, blah, blah (full list [here](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html))\n",
    "    + doesn't include the following languages: `R`\n",
    "        + just sayin :troll:\n",
    "\n",
    "let's talk about the `cli` and the `python sdk`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `aws` command line interface (`cli`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### what is the `aws cli`?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the phrase \"command line interface\" is always suggesting two things:\n",
    "\n",
    "1. we are using this on a command line (`tty`, `terminal.app`, `cmd`, `powershell`)\n",
    "1. this thing is an *interface* to some other system or set of tools\n",
    "\n",
    "in this case, amazon has kindly prepared a set of tools for you to use *on the command line* to *interface* with their various services (`ec2`, `s3`, `emr`, `rds`, *etc*)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the important thing to remember is that under the hood, `aws` (and `google cloud`, and `azure`) services are all basically `REST`ful `api` applications. any software which interacts with them will do so via `http` `request`s, and will implement some functions that replicate some subset of the various `REST` endpoints."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "take `ec2` for example. the \"things that you can do\" with `ec2` correspond to a long list of [`REST`ful `api` endpoints](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_Operations.html). clicking around in the console is equivalent to using your browser to build and submit `GET` and `POST` requests to these `api`s.\n",
    "\n",
    "`aws ec2` brings many of those endpoints into your command line environment. for example, the command\n",
    "\n",
    "```sh\n",
    "aws ec2 describe-instances\n",
    "```\n",
    "\n",
    "is a `bash`-ready `cli` implementation of the [DescribeInstances](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstances.html) endpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "[many/most `aws` services](https://docs.aws.amazon.com/cli/latest/reference/#available-services) have corresponding command (e.g. `aws s3`, `aws ec2`), and the various actions available"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### why use the `cli`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "there are many reasons you may wish to use the `cli`, but the primary motivations are that\n",
    "\n",
    "1. it is possible to script and automate actions with the `cli`, and not so easy (or possible) via the web console\n",
    "2. it can be called from within `shell` scripts\n",
    "3. the authentication process is different and -- depending on your perspective -- less onerous\n",
    "4. the interface from one service to the next is actually more consistent than the web console\n",
    "5. you can plug it in to other command line tools (e.g. your data science pipeline process!)\n",
    "6. I'm telling you you should and I have great authority\n",
    "7. it's in the next homework"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### an example\n",
    "\n",
    "maybe it would help to understand the sorts of things you might want to do with the `cli`. recall the webscraping we did of the webpage of the Connecticut power company (Eversource) where they posted [their reported outages](https://outagemap.eversource.com/external/default.html) on a webpage as a `json` request.\n",
    "\n",
    "when I actually used that data, I started downloading those files every 15 minutes, and I used an `aws` `ec2` instance to do that download. I actually saved those files to that machine, but I *could* have pushed them to `s3` instead -- it would have been as easy as\n",
    "\n",
    "```bash\n",
    "aws s3 cp outage.json s3://data.awesomeco.com/eversource/outages/\n",
    "```\n",
    "\n",
    "at the terminal prompt. the `cli` exposes every service as they were a simple linux command -- that's pretty cool."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### installing the `cli`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "one important thing to be aware of: `aws` released version 2 of their command line interface tool back in February of 2020.\n",
    "\n",
    "many systems you encounter in the real world will still have v1 of the `cli` installed on them, even though the current recommendation is to install v2. it is not yet fully adopted.\n",
    "\n",
    "there are some small differences between the two versions, but for the most part you are fine to learn one and then use the other without much issue.\n",
    "\n",
    "in this class, we will follow the aws recommendations and install v2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "version 2 of the `cli` is available for all platforms (download page [here](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)), but we will all be using it from our beautiful `ec2` instances.\n",
    "\n",
    "\n",
    "in the linux and mac world, you install the `cli` via `curl`-ing and running shell commands. let's do that!\n",
    "\n",
    "first thing's first: log in to your `ec2` instance now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "you may already have the `cli` installed (given that we picked the aws-maintained free tier ubuntu `ami`). let's check:\n",
    "\n",
    "```bash\n",
    "which aws\n",
    "```\n",
    "\n",
    "if the `cli` is installed, you will see a path here. if it is not, you will see no response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini exercise: everyone installs `aws`</div>**\n",
    "\n",
    "assuming the result of the above was a null response, we need to install the `cli`. we will do that by following [the instructions on the `cli` documentation page](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html):\n",
    "\n",
    "```bash\n",
    "# install unzip, used to... unzip\n",
    "sudo apt install -y unzip\n",
    "\n",
    "# download the cli\n",
    "curl \"https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip\" -o \"awscliv2.zip\"\n",
    "unzip awscliv2.zip\n",
    "sudo ./aws/install\n",
    "\n",
    "# confirm it is installed and see the version\n",
    "aws --version\n",
    "\n",
    "# clean up\n",
    "rm -r ./aws\n",
    "rm -r awscliv2.zip\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### getting help\n",
    "\n",
    "first, let's understand the syntax:\n",
    "\n",
    "```\n",
    "aws [options] <command> <subcommand> [parameters]\n",
    "```\n",
    "\n",
    "+ `options` affect the way the `aws` `cli` is working and are valid for all commands\n",
    "+ `command`s are (basically) the various service names (`ec2`)\n",
    "+ `subcommand`s are the various actions possible in that service\n",
    "+ `parameters` are the ways of parameterizing those subcommand actions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "obviously, google and stack overflow are your friends. in a pinch, though, you can toss in a `help` at any level in that hierarchy:\n",
    "\n",
    "```sh\n",
    "aws help\n",
    "aws ec2 help\n",
    "aws ec2 describe-instances help\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "take the `ec2` example above. the \"things that you can do\" with `ec2` correspond to a long list of [`REST`ful `api` endpoints](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_Operations.html). clicking around in the console is equivalent to building and submitting `GET` and `POST` requests to these `api`s.\n",
    "\n",
    "`aws ec2` brings many of those endpoints into your command line environment. check out\n",
    "\n",
    "```sh\n",
    "# q to quit\n",
    "aws ec2 help\n",
    "```\n",
    "\n",
    "to see the very many commands (endpoints) available to us."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "for example, look at one of them:\n",
    "\n",
    "```sh\n",
    "aws ec2 describe-instances help\n",
    "```\n",
    "\n",
    "and compare it to the [DescribeInstances](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstances.html) endpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">PAUSE FOR ZOOM BREAK</div>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### authentication"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "*every* `aws` action requires authentication, and the `cli` is no different. the game is the same here as it is elsewhere:\n",
    "\n",
    "1. you create access keys (authentication tokens) that are tightly controlled secrets\n",
    "1. you have \"service accounts\" or otherwise automatable actions \"sign in\" using those access keys, which you are responsible for storing in a way your programs can read and understand.\n",
    "\n",
    "this is very similar to what we did when you used my `google` authentication key for the `christopher_walkin` exercise. your code uses that secret to demonstrate that it's trusted by you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "wayyyyy back in lecture 1, when we created our non-root users in `iam`, you were given a prompt to save your credentials. you were given the option to *download a csv* with some secret information in it, including an access key id and access key value for your account.\n",
    "\n",
    "you still have that right?\n",
    "\n",
    "**right?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<strong><div align=\"center\">mini-exercise: access key creation</div></strong>\n",
    "<strong><div align=\"center\">https://console.aws.amazon.com/iam/home?region=us-east-1#/users</div></strong>\n",
    "<em><div align=\"center\">make sure to save the access key value somewhere secure -- you can't ever get it again</div></em>\n",
    "\n",
    "each `iam` user has the ability to have up to two access keys. to create one for yourself:\n",
    "\n",
    "1. head over to the [`iam` `users` dashboard](https://console.aws.amazon.com/iam/home?region=us-east-1#/users).\n",
    "1. select the `gu511` account we created in the `iam` section.\n",
    "1. click on the \"Security credentials\" tab\n",
    "1. click on the \"Create access key\" button\n",
    "1. download this file to a place you will remember\n",
    "    1. **you can't ever get this particular \"secret access key\" again**\n",
    "    1. you can create other access keys, so it's not the literal end of the world\n",
    "    1. ok, even if you couldn't, it's still not the literal end of the world. no need for hyperbole\n",
    "1. click \"ok\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "ok. at this point we should all have:\n",
    "\n",
    "1. an access key ID (recoverable at any time from the `iam` console\n",
    "2. a secret access key (if you don't have it, you done goofed, and have to make a new key)\n",
    "\n",
    "so how do we *use* this access key?\n",
    "\n",
    "let's log in to our `ec2` server and try it out!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the first thing we do is run the `aws configure` command to add our access key information.\n",
    "\n",
    "check out `aws configure help`, sepcifically the `EXAMPLES` section\n",
    "\n",
    "```\n",
    "EXAMPLES\n",
    "       To create a new configuration:\n",
    "\n",
    "          $ aws configure\n",
    "          AWS Access Key ID [None]: accesskey\n",
    "          AWS Secret Access Key [None]: secretkey\n",
    "          Default region name [None]: us-west-2\n",
    "          Default output format [None]:\n",
    "\n",
    "       To update just the region name:\n",
    "\n",
    "          $ aws configure\n",
    "          AWS Access Key ID [****]:\n",
    "          AWS Secret Access Key [****]:\n",
    "          Default region name [us-west-1]: us-west-2\n",
    "          Default output format [None]:\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "this tool is used to store access tokens so that you can authenticate the `aws` commands as being \"by `gu511`\"\n",
    "\n",
    "**best practice note**: `cli` supports a concept called \"profiles\" -- you could add more than one set of access keys to a single `ec2` instance and user account\n",
    "\n",
    "for example, suppose you have one `ec2` server for generic web-scraping or etl work, but several different client projects. you could create a separate \"profile\" for each project (`scrape`, `the-good-client`, `the-bad-client`, `the-ugly-client`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini-exercise: verify `cli` permissions are correct</div>**\n",
    "\n",
    "let's create a profile `gu511` and enter the access key credentials we just generated and downloaded for user `gu511` in `iam`.\n",
    "\n",
    "1. on your local machine, open the `csv` we downloaded with `gu511`'s access key info\n",
    "1. on your `ec2` instance, run `aws configure --profile gu511`\n",
    "1. fill out the interactive prompts\n",
    "    1. the first two (`AWS Access Key ID` and `AWS Secret Access Key`) are in your `csv`\n",
    "    1. for the `Default region name`, enter `us-east-1`\n",
    "    1. for the `Default output format`, press enter (leave it blank)\n",
    "\n",
    "run the following from your `ec2` server after creating your profile\n",
    "\n",
    "```bash\n",
    "aws --profile gu511 ec2 describe-instances\n",
    "```\n",
    "\n",
    "this should put you in the default `less` file viewer and you should see a big, sloppy mess of info from this as a `json` blob -- if not, drop in the zoom room and let's debug!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "although I have gone on at length about how un-recoverable the access keys are, that's actually not true -- as soon as you've run `aws configure` one time, they're actually saved in plain text on your file system: not great!!\n",
    "\n",
    "check out\n",
    "\n",
    "```bash\n",
    "less ~/.aws/credentials\n",
    "```\n",
    "\n",
    "you should see something like\n",
    "\n",
    "```\n",
    "[gu511]\n",
    "aws_access_key_id = [YOUR ACCESS KEY ID HERE]\n",
    "aws_secret_access_key = [YOUR ACCESS KEY VALUE HERE]\n",
    "```\n",
    "\n",
    "what do you think of this? how secure is or isn't this? who can see this?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "there is also a file storing configuration information independent of the credentials -- let's look at the contents of that file\n",
    "\n",
    "check out\n",
    "\n",
    "```bash\n",
    "less ~/.aws/config\n",
    "```\n",
    "\n",
    "you should see something like\n",
    "\n",
    "```\n",
    "[profile gu511]\n",
    "region = us-east-1\n",
    "```\n",
    "\n",
    "(press `q` to quit the reader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "what we just did was set the default access credentials for a `profile` named `gu511`. this means that any time we run\n",
    "\n",
    "```sh\n",
    "aws --profile gu511 [command] [subcommand]\n",
    "```\n",
    "\n",
    "we will be submitting the access key info we just added to that profile allong with our request to `aws` to run `command subcommand`.\n",
    "\n",
    "if we *don't* add the `profile`, the `aws` command will still ask `aws` to perform `command subcommand`, but it won't have any information on hand to send to authenticate. we will see an error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini-exercise: see the authentication error if you try to use `aws` without the `gu511` profile</div>**\n",
    "\n",
    "run the following from your `ec2` server\n",
    "\n",
    "```bash\n",
    "aws ec2 describe-instances\n",
    "```\n",
    "\n",
    "you should see the following error\n",
    "\n",
    "```\n",
    "Unable to locate credentials. You can configure credentials by running \"aws configure\".\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "at any time, you are welcome to run *just*\n",
    "\n",
    "```sh\n",
    "aws configure\n",
    "```\n",
    "\n",
    "(i.e. the command we ran about but without the `--profile` flag) to set your default credentials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### the good news and the bad news"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the good news is that we just\n",
    "\n",
    "1. installed the `aws cli`\n",
    "2. figured out how to authenticate it using access keys\n",
    "3. successfully executed a test command\n",
    "\n",
    "the bad news is that step 2 -- using access keys to authenticate -- is neither the easiest nor best (depending on your use case) way of authenticating.\n",
    "\n",
    "the reason this is so is that it is possible to [assign an `iam role` to your `ec2` server](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html) and have all `aws cli` calls on that server authenticate through that role."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "so why did I drag you all through this? sadism?\n",
    "\n",
    "that statement above -- \"depending on your use case\" -- should be a tell-tale sign."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "access keys are a complication, so they must either be solving a specific problem, or they are totally worthless. in this case, it's the former. you can only grant one role -- and one set of permissions -- to a single `ec2` instance. for you right now, that might make more sense: you're one person with one server and you feel you should have the same admin-level access to everything you own.\n",
    "\n",
    "in the real world, you may need better, more granular security settings, and this is where the profiles, separate users, and separate authentication methods come in."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## let's set up that sweet `iam role` thing, then!\n",
    "\n",
    "this is relatively straightforward, so let's walk through it together:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "+ create the `iam role`\n",
    "    1. navigate to your `iam` dashboard in the `aws` web console.\n",
    "    1. click on `roles` on the left hand menu\n",
    "    1. click \"Create role\"\n",
    "    1. with the default \"AWS Service\" trusted entity block selected, choose the \"common use case\" of `ec2`, then click \"Next: Permissions\"\n",
    "    1. for now let's just give it full access to our `ec2` instances and `s3` buckets\n",
    "        1. in the Filter search bar, type \"S3Full\"\n",
    "        1. check the box for\n",
    "        1. repeat with \"EC2Full\"\n",
    "    1. click the \"Next: Tags\" and then the \"Next: Review\" buttons\n",
    "    1. choose whatever name and description you want (e.g. `ec2_s3_full_access`)\n",
    "    1. click \"Create Role\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "+ assign it to our `ec2` instance\n",
    "    1. navigate to your `ec2` dashboard in the `aws` web console\n",
    "    2. click on `instances` on the left hand menu\n",
    "    3. right click on your `ec2` instance\n",
    "    4. select \"modify iam role\"\n",
    "    5. select the role you just created and save\n",
    "    6. you're done!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "now, when we make `aws` calls, we shouldn't need to authenticate *at all*, which is pretty great. we still *could* use the authentication we defined under profile `gu511`, but now that the role is attached, if we *don't* provide credentials explicitly, we will get the permissions of the `iam` role.\n",
    "\n",
    "however, when we ran `aws configure --profile gu511`, we did more than just enter credentials: we also set a default `region` value. while future calls to `aws` without the `--profile gu511` flag will now have permissions regardless of access keys, they won't have a default region set, and we will see errors as a result of this\n",
    "\n",
    "for example, run\n",
    "\n",
    "```sh\n",
    "aws ec2 describe-instances\n",
    "```\n",
    "\n",
    "and see what happens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "if we want to use our default profile, we should still add a default `region` to it. run\n",
    "\n",
    "```sh\n",
    "# no profile name\n",
    "aws configure\n",
    "```\n",
    "\n",
    "and for both the first two Access key fields, let them be blank (just press enter). update the `region` to have a value `us-east-1`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "after you've done this, check out the contents of `~/.aws/config` again. you should see something like\n",
    "\n",
    "```\n",
    "[default]\n",
    "region = us-east-1\n",
    "[profile gu511]\n",
    "region = us-east-1\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "now, what happens when we try to execute our `ec2` query again:\n",
    "\n",
    "```bash\n",
    "aws ec2 describe-instances\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "that's pretty *NOT BAD*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">PAUSE FOR ZOOM BREAK</div>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `python sdk`, *aka* `boto`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "under the hood, the `aws` command is actually implemented in `python` -- that is, whenever you type `aws [command] [subcommand]`, you are running a `python` program that is making `REST` calls out to the various aws `api`s that you are using.\n",
    "\n",
    "that `python` code is of course open source, and available for you to use in your own programs. the package is named `boto3`, where \"boto\" is the name of a species of fresh-water dolphin that commonly swim in rivers in portuguese-speaking South American countries (such as the Amazon river, in Brazil)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "we will practice using the `boto3` library up close in a later lecture (on `s3`), but for now, let's just replicate the `ec2` information request we made with the `cli`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### install `boto3`\n",
    "\n",
    "the first step, as usual, is installing the software. let's start by attempting to `conda` install `boto3`:\n",
    "\n",
    "```bash\n",
    "conda install boto3\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "now, if we *hadn't* just gone through the process of authentication, we would need to do that all over again. as it stands, though, we are all set with that.\n",
    "\n",
    "see how I just stacked the deck to make it seem like `python` was sooooooo easy? how unfair! how biased!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## `boto3` organization\n",
    "\n",
    "I've said it now several times: somewhere, deep down, `aws` `cli` and `boto3` are just making `http` requests to `aws` `REST`ful `api` endpoints.\n",
    "\n",
    "`boto3` is no different, but its place in the richer (than `bash`) `python` environment gives it the ability to be a bit more powerful. there are three types of objects we should know about."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### `session`\n",
    "\n",
    "as we saw in the `http` lecture, a `session` is an object often built in client-server interaction programs which encapsulates various things you want to remember and re-use every time you make a request -- for example, your configuration parameters and credentials.\n",
    "\n",
    "**you don't have to create a session**, but you may want to: it allows you to skip specifying configuration parameters every time you create a `client` or `resource` (more on those below).\n",
    "\n",
    "```python\n",
    "# set parameters\n",
    "session = boto3.session.Session(region_name='us-east-1')\n",
    "\n",
    "# set a whole dang profile\n",
    "session = boto3.session.Session(profile_name='gu511')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "*note*: if you have a default configuration in the configuration file at `~/.aws/credentials`, this may be redundant. or, this may be an easy way of stipulating a profile name once and only once per script -- it's up to you!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### `client`\n",
    "\n",
    "`boto3` has two different paradigms for calling out to the `aws` services:\n",
    "\n",
    "1. `client`s\n",
    "1. `resource`s\n",
    "\n",
    "we will talk about `client`s first, as they are much more conceptually straight-forward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the `client` objects available in `boto3` are `python` objects created as direct, one-to-one implementations of the `REST` `api` endpoints for most services. they are auto-generated and fully functional, but a little bit more low-level and clunky than their object-oriented counterparts (`resource`s, below).\n",
    "\n",
    "basically, you should use `client` objects if you can't figure out how to use the `resource` objects, or there isn't a corresponding `resource`. I wouldn't say \"never\", but that is clearly the intention of the module architects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini-exercise: obtain our `ec2` information using `boto3 clients`</div>**\n",
    "\n",
    "let's duplicate our `ec2` example from above:\n",
    "\n",
    "```python\n",
    "import boto3\n",
    "\n",
    "session = boto3.session.Session(profile_name='gu511')\n",
    "\n",
    "ec2 = session.client('ec2')\n",
    "\n",
    "insts = ec2.describe_instances()\n",
    "\n",
    "print(insts)\n",
    "\n",
    "# note: I may have to choose the index after Instances\n",
    "# because I have multiple instances\n",
    "print('\\nand look what we have here:')\n",
    "print(insts['Reservations'][0]['Instances'][0]['IamInstanceProfile'])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### `resource`\n",
    "\n",
    "the `boto3` architects have created more object-oriented interfaces to the different service `api`s. the motivation, as best I can tell, is to try and create a normalized, shared interface for all services (even though the apis themselves are not necessarily similar).\n",
    "\n",
    "in any case, `resource` implementations are recommended, and will often be \"more pythonic\" in syntax (e.g. favoring iteration and dot notation)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "since 2018, only the following 10 `aws` services are supported with `resource` object interfaces:\n",
    "\n",
    "+ cloudformation\n",
    "+ cloudwatch\n",
    "+ dynamodb\n",
    "+ ec2\n",
    "+ glacier\n",
    "+ iam\n",
    "+ opsworks\n",
    "+ s3\n",
    "+ sns\n",
    "+ sqs\n",
    "\n",
    "(to see this list at any time, run `session.get_available_resources()`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini-exercise: obtain our `ec2` information using `boto3 resources`</div>**\n",
    "\n",
    "let's duplicate our `ec2` example from before:\n",
    "\n",
    "```python\n",
    "import boto3\n",
    "\n",
    "session = boto3.session.Session(profile_name='gu511')\n",
    "\n",
    "ec2 = session.resource('ec2')\n",
    "\n",
    "for (i, instance) in enumerate(ec2.instances.all()):\n",
    "    print(f'instance {i} has iam profile: {instance.iam_instance_profile}')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "note the \"pythonic\" differences here: via the `aws cli` and `boto3` `client` methods, we get a structured `json` object. as one example, the `iam` instance profile is accessible as\n",
    "\n",
    "```python\n",
    "insts['Reservations'][0]['Instances'][0]['IamInstanceProfile']\n",
    "```\n",
    "\n",
    "in the `boto3 resource` paradigm, we instead get\n",
    "\n",
    "1. an `ec2` resource object\n",
    "2. an iterator over `ec2` instances\n",
    "3. individual instance objects\n",
    "4. instance objects have attributes, like `iam_instance_profile`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">PAUSE FOR ZOOM BREAK</div>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### general workflow\n",
    "\n",
    "the general workflow when using the `boto3` library is as follows:\n",
    "\n",
    "+ *optional*: create a \"session\" object to encapsulate connection and credential info\n",
    "+ create a `resource` object\n",
    "    + will likely have, as members, `collection` iterables\n",
    "    + may need to authenticate on the spot (if no `config` file exists) or specify region (if no `session`)\n",
    "    + example service types: `ec2`, `iam`, `s3`, `rds`\n",
    "+ use the member functions of that service object to perform standard tasks\n",
    "    + call `help(serviceobject)` to find out more\n",
    "+ when all else fails, try a direct `client` object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<em><strong><div align=\"center\">the majestic `boto` in its natural habitat, the `amazon`</div></strong></em>\n",
    "<div align=\"center\"><img src=\"https://ferdinandodesousa.files.wordpress.com/2017/11/boto-do-araguaia.jpg\" width=\"700px\"></div>\n",
    "\n",
    "# END OF LECTURE\n",
    "\n",
    "next lecture: [AWS `s3` (simple storage service)](010_s3.ipynb)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}