Skip to content

Commit

Permalink
Merge pull request #184 from tinnightcap/develop
Browse files Browse the repository at this point in the history 
Updates for new nubis-deploy docker image
  • Loading branch information
@limed
limed authored Aug 29, 2017
2 parents 4a3ee47 + 5df1a4d commit 5f45fb2
Show file tree
Hide file tree
Showing 2 changed files with 167 additions and 83 deletions.
201 changes: 118 additions & 83 deletions PREREQUISITES.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,35 @@
# Prerequisites

Before you can contribute to the Nubis project, you'll need to have a few
tools installed.
tools installed and configured.

## Configure shell

When your user is added to an account you will receive an encrypted email
containing The necessary information for you to access the account.

The first thing I like to do is set up some local shell variables to make the
following steps a bit simpler. Of course you will **need to replace** the
```ACCOUNT_NAME```, ```ACCOUNT_NUMBER``` and ```LOGIN``` with the ones you
received in the user credentials email.

```bash

ACCOUNT_NAME='nubis-training'; ACCOUNT_NUMBER='517826968395'; LOGIN='<login>'

```

## GitHub Account

Nubis is entirely hosted on github, and we derive most of our workflows from
Nubis is entirely hosted on github, and we derive most of our work-flows from
GitHub's recommended practices.

If you are new to git or GitHub, you are probably better familiarizing yourself
with that first. There are a lot of handy tutorials straight from the source
[here](https://www.atlassian.com/git/tutorials/)
If you are new to git or GitHub, you are probably better off familiarizing
yourself with that first. There are a lot of handy tutorials straight from the
source [here](https://www.atlassian.com/git/tutorials/)

First you will need to set up an account on GitHub. It doesn't have to be
anything special, any old account will do. To set up a GitHub account,
click [here](https://github.com/join).
You will need to set up an account on GitHub. It doesn't have to be anything
special, any old account will do. To set up a GitHub account, click [here](https://github.com/join).

You will also want to set up your ssh keys with GitHub. You can do that [here](https://github.com/settings/ssh).

Expand All @@ -34,11 +49,20 @@ aptitude install git

```

Homebrew users:

```bash

brew install git

```

## AWS CLI

Next, you need to install the AWS CLI tool. You can install it by following the
instructions at the top of [this page](http://aws.amazon.com/cli/). For Mac and
Linux users you can simply:
instructions at the top of [this page](http://aws.amazon.com/cli/).

For Linux (and some mac) users:

```bash

Expand All @@ -54,29 +78,17 @@ brew install awscli

```

## Terraform (0.6.16+)

Get it from [Terraform.io](https://www.terraform.io/downloads.html). We use
Terraform for deploying everything from the account to the application. If you
are interested in the decision to use Terraform over Cloudformation you can read
about it [here](./TEMPLATING.MD).

It's a simple Go binary bundle, just unzip and drop in your $PATH

Make sure you obtain version 0.8.8.

Try [this path](https://releases.hashicorp.com/terraform/0.8.8/).

## AWS Credentials

In order to work with AWS you will need to set up some credentials. This is a
somewhat involved process as all access to AWS requires utilizing a multi-factor
authentication (MFA) device. When your user is added to an account you will
receive an encrypted email containing a key pair.

NOTE: These keys need to remain secret. You need to take the utmost care; DO NOT
check them into git, send them via unencrypted email, copy them into a pastebin,
etcetera.
**NOTE:** These keys need to remain secret. You need to take the utmost care;
**DO NOT** check them into git, send them via unencrypted email, copy them into
a pastebin, etcetera.


### aws-vault

Expand All @@ -85,7 +97,7 @@ AWS API credentials. You will need to download this tool and place it on your
path.

Once installed you will use the aws-vault tool to authenticate for all access
and actions within AWS. Fist you will need to set up your MFA device.
and actions within AWS.

Lets start by making sure aws-vault is installed and working correctly:

Expand All @@ -95,7 +107,7 @@ aws-vault --version

```

This should return something like ```v3.3.0```.
This should return something like ```v3.7.1```.

If you are using linux you need to set your backend to use kwallet. I recommend
placing this in one of your startup scripts, say ```~/.bashrc```:
Expand All @@ -106,17 +118,6 @@ export AWS_VAULT_BACKEND=kwallet

```

The next thing I like to do is set up some local shell variables to make the
following commands a bit simpler. Of course you will **need to replace**
the ```ACCOUNT_NAME```, ```ACCOUNT_NUMBER``` and ```LOGIN``` with the ones you
received in the user credentials email.

```bash

ACCOUNT_NAME='nubis-training'; ACCOUNT_NUMBER='517826968395'; LOGIN='jcrowe'

```

Now you can run the aws-vault command to set up the account. This will ask you
for the 'Access Key ID' and the 'Secret Access Key'. You will need to get those
from the user credentials email as well:
Expand Down Expand Up @@ -150,47 +151,47 @@ aws iam create-virtual-mfa-device \
```

You should see output similar to the following. The number here should
correspond to the account number and the 'jcrowe' part should be your user-name
(not mine ;-D):
correspond to the account number and the ```<login>``` portion of the serial
number should be your user-name:

```bash

{
"VirtualMFADevice": {
"SerialNumber": "arn:aws:iam::517826968395:mfa/jcrowe"
"SerialNumber": "arn:aws:iam::517826968395:mfa/<login>"
}
}

```

You will need to view the ${LOGIN}.png file and use it to configure your MFA
application. If you have imagemagic installed you can try ```display$LOGIN.png```
application. If you have imagemagic installed you can try ```display $LOGIN.png```
or just open it with any image viewer. I use the [duo mobile app](https://duo.com/solutions/features/two-factor-authentication-methods/duo-mobile),
however the [google authenticator app](https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2&hl=en)
works as well. Really, most MFA apps work here so if you have one you already
use or prefer it will probably be just fine.
use or prefer it should be just fine.

To finish up, you need to enable the mfa device. This step is basically proving
that you are you and everything is configured correctly. You will need to
provide two sequential MFA codes. You get these codes from the MFA application
you just set up.

NOTE: The codes must be sequential and entered in the correct order in the
following command. (Replace ```123456``` and ```654321``` with codes from
your app):
**NOTE:** The codes must be sequential and entered in the correct order in the
following command. (Replace ```<code 1>``` and ```<code 2>``` with codes from
your mfa app):

```bash

aws-vault exec -n ${ACCOUNT_NAME} -- \
aws iam enable-mfa-device \
--user-name ${LOGIN} \
--serial-number arn:aws:iam::${ACCOUNT_NUMBER}:mfa/${LOGIN} \
--authentication-code-1 123456 \
--authentication-code-2 654321
--authentication-code-1 <code 1> \
--authentication-code-2 <code 2>

```

You need to configure your AWS CLI tools to make use of the virtual MFA device.
You need to configure the AWS CLI tools to make use of the virtual MFA device.
You can either add this to your ```~/.aws/config``` file manually or run the
following bash snippet:

Expand Down Expand Up @@ -220,7 +221,7 @@ To test that everything has been set up correctly, run the following command:

```bash

aws-vault exec ${ACCOUNT_NAME}-ro -- aws ec2 describe-regions
vault-exec aws ec2 describe-regions

```

Expand All @@ -229,7 +230,7 @@ entering it you should see some output similar to this:

```bash

Enter token for arn:aws:iam::517826968395:mfa/jcrowe: 123456
Enter token for arn:aws:iam::517826968395:mfa/<login>: <code>
{
"Regions": [
~snip~
Expand Down Expand Up @@ -258,16 +259,16 @@ aws-vault --debug login ${ACCOUNT_NAME}-ro

## nubis-builder

This is a Docker image we built to drive Packer. The installed tools greatly
simplify Packer configuration. It's fairly simple to install and comes with all
This is a Docker image we built to drive Packer. The embedded tools greatly
simplify Packer configuration. It's quite simple to install and comes with all
of the required dependencies.

### Install docker and required libraries

#### Linux

NOTE: These packages are not required, but Docker will have reduced performance
if they are not installed.
**NOTE:** These packages are not required, but Docker will have reduced
performance if they are not installed.

```bash

Expand All @@ -277,7 +278,7 @@ sudo apt-get install \

```

Install docker
Install docker:

```bash

Expand All @@ -286,7 +287,7 @@ sudo apt install docker.io
```

This section can be ignored, however you will need to perpend 'sudo' to all
docker commands that follow.
docker commands (including adjusting the bash alias above).

```bash

Expand All @@ -295,7 +296,7 @@ sudo usermod -aG docker $USER

```

Log out & back in to pick up new group membership
Log out & back in to pick up your new group membership.


#### Mac:
Expand All @@ -304,6 +305,9 @@ Log out & back in to pick up new group membership

### Test that Docker is set up and running correctly

Try running the hello-world docker image. This will download and run the docker
image:

```bash

docker run hello-world
Expand All @@ -312,7 +316,23 @@ docker run hello-world

### Run the nubis-builder Docker image

Clone the nubis-skel repository
Lets start by setting up a few shell alias to make these commands a bit more
manageable:

```bash

alias vault-exec="aws-vault exec ${ACCOUNT_NAME}-admin --"

NUBIS_DOCKER=( 'docker' 'run' \
'-u' "$UID:$(id -g)" \
'--interactive' \
'--tty' \
'--env-file' "$(echo ~)/.docker_env" \
'-v' "$PWD:/nubis/data" )

```

Clone the nubis-skel repository:

```bash

Expand All @@ -321,7 +341,7 @@ cd nubis-skel

```

Set up docker variables file
Set up a docker variables file:

```bash

Expand All @@ -335,29 +355,28 @@ EOH

```

Fire off the nubis-builder docker image
Fire off the nubis-builder docker image. This will create an AWS AMI for you to
deploy:

```bash

aws-vault exec ${ACCOUNT_NAME}-admin -- \
docker run -u $UID:$(id -g) -it \
--env-file ~/.docker_env \
-e GIT_COMMIT_SHA=$(git rev-parse HEAD) \
-v $PWD:/nubis/data nubisproject/nubis-builder:v0.4.0
vault-exec "${NUBIS_DOCKER[@]}" -e GIT_COMMIT_SHA=$(git rev-parse HEAD) nubisproject/nubis-builder:v0.4.0

```

Create terraform.tfvars from terraform.tfvars-dist and change values to these settings:
Create a terraform.tfvars from terraform.tfvars-dist and change values to these
settings (Be sure to replace all of the ```<variables>``` whit appropriate ones
for your application):
- account = "nubis-training"
- region = "us-west-2"
- environment = "stage"
- service_name = "<username>-skel"
- ssh_key_name = "<username>-key"
- ssh_key_file = "/<path to your ssh key>/.ssh/id_rsa.pub"
- nubis_sudo_groups = "<value provided by Nubis team>"
- nubis_user_groups = "<value provided by Nubis team>"
- service_name = "```<username>```-skel"
- ssh_key_name = "```<username>```-key"
- ssh_key_file = "/```<path to your ssh key>```/.ssh/id_rsa.pub"
- nubis_sudo_groups = "```<value provided by Nubis team>```"
- nubis_user_groups = "```<value provided by Nubis team>```"

The service_name and ssh_key_name must be unique in the Nubis account but otherwise do not matter.
The service_name and ssh_key_name **must** be unique in the Nubis account.

```bash

Expand All @@ -366,25 +385,41 @@ vim nubis/terraform/terraform.tfvars

```

Deploy nubis-skel to the training account
Deploy nubis-skel to the training account. Start by executing a plan to make
sure of the changes you will be making. Next apply the plan, this will deploy
the nubis-skel application into the account.

```bash

aws-vault exec ${ACCOUNT_NAME}-admin -- terraform get -update=true nubis/terraform
aws-vault exec ${ACCOUNT_NAME}-admin -- \
terraform plan \
-var-file=nubis/terraform/terraform.tfvars nubis/terraform
vault-exec "${NUBIS_DOCKER[@]}" nubisproject/nubis-deploy:v0.2.0 plan
vault-exec "${NUBIS_DOCKER[@]}" nubisproject/nubis-deploy:v0.2.0 apply

```

Test your deployment by navigating to the ELB endpoint that is in the outputs
section. It should look something like this:

```bash

Outputs:

address = https://www.<app-name>.stage.<region>.<account-name>.nubis.allizom.org/
Outputs:

```

To finish up, remove your application from the training account:

```bash
aws-vault exec ${ACCOUNT_NAME}-admin \
-- terraform apply \
-var-file=nubis/terraform/terraform.tfvars nubis/terraform
vault-exec "${NUBIS_DOCKER[@]}" nubisproject/nubis-deploy:v0.2.0 destroy
```

## Fin

That should be all you need to get started. If you run into any issue or have
any trouble, please reach out to us. We are happy to help and are quite
interested in improving the project in any way we can. We are on irc.mozilla.org
interested in improving the project in any way we can. We are on Slack
in #nubis-users or you can reach us on the mailing list at
nubis-users[at]googlegroups.com
Loading

0 comments on commit 5f45fb2

Please sign in to comment.