Create shared dock ami #611

thejsj · 2017-03-13T00:31:42Z

What this PR does

Makes AMIs environment/customer agnostic (does not bake certs to AMI or any environment variables)
Create script to automatically generate launch configurations
Add new tags that don't need to be run when creating docks
Remove presumption that Rollbar key has to be present
Updates vault port to a variable.
Remove registrly.runnable.com for image-builder
Upgrades krain

Bugs/Things that need to be fixed

Docker needs to be run without TLS when pulling images because default docker (from service) does not have the necessary certs to run. You might need to manually deal start it.

How to create environment agnostic AMI

Run base.yml with these flags.

ansible-playbook
  --vault-password-file /Users/hiphipjorge/.vaultpass
  --extra-vars "host=$DOCK_IP"
  --skip-tags=ensure_registry,loggly
  -e restart=true
  base.yml

Run through mounting docker as you normally would. Because docker needs to be setup for the next step, you need to go into /etc/default/docker and remove TLS from it. In the future this should be done automatically.
Run dock.yml with following arguments:

ansible-playbook 
  --vault-password-file /Users/hiphipjorge/.vaultpass
  --extra-vars "dock=$DOCK_IP"
  --skip-tags vault_files,consul_values,render_node_service_config,ensure_registry,loggly
  dock.yml

Create launch configuration for environment:

ansible-playbook -i delta-hosts/ dock-generate-launch-config.yml

Tests

Create new AMI and launch configuration in gamma

Make sure there are no errors in /var/log/cloud-init-output.log
Make sure there are no errors in user-script-dock-init.log
Make sure Docker comes up docker ps
Make sure Krain comes up curl localhost:3100 -v and files show up
Make sure Charon is up (C2C works in environment)
Deploy docker-listener with changes to make sure it works

Create new AMI and launch configuration in hot grizzly

Make sure there are no errors in /var/log/cloud-init-output.log
Make sure there are no errors in user-script-dock-init.log
Make sure Docker comes up docker ps
Make sure Krain comes up curl localhost:3100 -v and files show up
Make sure Charon is up (C2C works in environment)
Deploy docker-listener with changes to make sure it works

Deployment (post-merge)

Ensure that all environments have the given changes.

deployed to gamma
deployed to delta

TODO for the future

Tag image builder with both runnable/image-builder and registry.runnable.com/runnable/image-buidler
Add mounting of /docker volume to script in order for everything to live in this script
Add base.yml to dock.yml in order to be able to build dock with one command
Handle installation of Loggly without keys (optional running)

…t require variables tied to environment

thejsj · 2017-03-13T01:01:49Z

Requires running command in following way:

ansible-playbook 
  --vault-password-file /Users/hiphipjorge/.vaultpass
  --extra-vars "dock=10.4.190.15" 
  --skip-tags vault_files,loggly,consul_values,render_node_service_config,ensure_registry
  dock.yml

thejsj · 2017-03-13T17:27:08Z

Requires running base.yml the following way:

ansible-playbook
  --vault-password-file /Users/hiphipjorge/.vaultpass
  --extra-vars "host=10.4.172.205"
  --skip-tags=ensure_registry,loggly
  base.yml

anandkumarpatel · 2017-03-13T18:26:08Z

ansible/dock-generate-launch-config.yml

@@ -0,0 +1,10 @@
+---
+- hosts: cream


should be localhost

The problem is that the whole reason we're doing this is to have access to variables tied to an environment, but I don't know how to run this on localhost while getting the variables for another environment...

add localhost to [grizzly:children] in hosts file

anandkumarpatel · 2017-03-13T18:27:32Z

ansible/roles/dock_launch_config/tasks/main.yml

@@ -0,0 +1,39 @@
+---
+- name: create new config file for krain


doesn't dock-init do this?

Not right now, no. Seems like it should though. Do we want to move it there? Would need some guidance on how to do this.

Actually, we can't do that because we start the services (krain and charon) before running dock-init.

where did you get these files from? why not just use/update the regular roles to install this? scared of something new

anandkumarpatel · 2017-03-13T18:27:48Z

ansible/roles/dock_launch_config/tasks/main.yml

+    - app_name: "krain"
+
+- name: encode krain config to base64
+  shell: cat ~/{{ app_name }}.conf | base64 -w 0


are you encoding this for a reason?

So that we don't have to deal with weird formatting/spacing issues, since we're just adding this to a file to echo it into a file.

why are we echoing instead of copying it directly to host? does not look like you are transforming this file at all

echo {{ krain_base64['stdout'] }} | base64 --decode >> /etc/init/krain.conf

no transform just a decode?

anandkumarpatel · 2017-03-13T18:28:14Z

ansible/roles/dock_launch_config/templates/init.tmpl

+# Set ENV files
+export CONSUL_PORT={{ consul_api_port }}
+export CONSUL_HOSTNAME={{ ansible_default_ipv4.address }}
+export VAULT_PORT=65240


should be {{ vault_port }}

anandkumarpatel · 2017-03-13T18:28:34Z

ansible/roles/dock_launch_config/templates/init.tmpl

+echo {{ charon_base64['stdout'] }} | base64 --decode >> /etc/init/charon.conf
+
+# Start services
+start amazon-ssm-agent


not needed for dock (only for hot pool)

anandkumarpatel · 2017-03-14T20:33:54Z

ansible/dock-generate-launch-config.yml

@@ -0,0 +1,10 @@
+---
+- hosts: cream


add localhost to [grizzly:children] in hosts file

anandkumarpatel · 2017-03-14T20:36:56Z

ansible/roles/dock-images/tasks/main.yml

-    insertafter: EOF
-    block: |
-      {{ registry_address }} registry.runnable.com
+  command: docker login -u {{ docker_hub_username }} -p {{ docker_hub_password }}


what is this too?

anandkumarpatel · 2017-03-14T20:41:22Z

ansible/roles/dock_launch_config/tasks/main.yml

@@ -0,0 +1,39 @@
+---
+- name: create new config file for krain


where did you get these files from? why not just use/update the regular roles to install this? scared of something new

anandkumarpatel · 2017-03-14T20:43:33Z

ansible/roles/dock_launch_config/tasks/main.yml

+    - app_name: "krain"
+
+- name: encode krain config to base64
+  shell: cat ~/{{ app_name }}.conf | base64 -w 0


why are we echoing instead of copying it directly to host? does not look like you are transforming this file at all

echo {{ krain_base64['stdout'] }} | base64 --decode >> /etc/init/krain.conf

no transform just a decode?

anandkumarpatel · 2017-03-14T20:53:31Z

ansible/roles/image-builder/tasks/main.yml

-  command: sudo docker build --no-cache --tag="registry.runnable.com/{{ image_builder_docker_namespace }}:{{ git_branch }}" /opt/runnable/image-builder
+  command: sudo docker build --no-cache --tag="{{ image_builder_docker_namespace }}:{{ git_branch }}" /opt/runnable/image-builder

 - name: push image-builder


delete we no longer need to do this

FYI, this requires a change in docker-listener for whitelisted image inspect list.

…ripts into create-shared-dock-ami * 'create-shared-dock-ami' of github.com:CodeNow/devops-scripts: Remove psad from dock (#613) Added s3 log bucket to ansible

…e-shared-dock-ami * 'master' of github.com:CodeNow/devops-scripts: Add aws host for API Moved cmd order Updated url Added clio port. Revert "Revert "Added clio http url env var"" Revert "Added clio http url env var" Added clio http url env var Remove mongo certs to run pheidi Added deploy song Added new DB users in gamma/delta Updated to point clio to the same DB as everyone else :( Removed quotes Fixed host Added mongo hosts. Point to api's db Removed extra host Removed dep on mongo-clio Downgraded to node 4 Add mongo-clio and clio to services.

…e-shared-dock-ami * 'master' of github.com:CodeNow/devops-scripts: add clio to worker and socket (#622) Remove khronos mongo certs (#621) remove redis tls (#620) split out delta-app to 2 boxes. remove unused configs (#619)

…tion of paths

anandkumarpatel · 2017-03-25T00:30:30Z

ansible/group_vars/alpha-docker-listener.yml

  -e DATADOG_HOST={{ datadog_host_address }}
  -e DATADOG_PORT={{ datadog_port }}
  -e DOCKER_CERT_PATH=/etc/ssl/docker
+  -e IMAGE_INSPECT_LIST=localhost,registry.runnable.com,runnable


this might be bad, we will now inspect all hemindal containers, can you make this runnable/image-builder? and test it works? also you can remove registry.runnable.com

anandkumarpatel · 2017-03-25T00:37:01Z

ansible/roles/dock_launch_config/templates/upstart.conf

@@ -0,0 +1,41 @@
+#!upstart
+description "{{ app_name }}"
+author "Anandkumar Patel"


you made this, I am not taking the blame for its bugs

haha No problem!

anandkumarpatel · 2017-03-25T00:38:52Z

ansible/roles/git_repo/tasks/main.yml

    msg: "application Installed: {{ app_name }}, branch : {{ git_branch }} "

 - name: create repository dir
+  become: true


worst. we were trying to remove all the places we do this.... but if you have to that is fine

I forget why I had to this... but I feel there was a reason for it.

yea, because the /opts dir was made with sudo. its ok

anandkumarpatel · 2017-03-25T00:44:15Z

ansible/roles/base_ubuntu/tasks/main.yml

@@ -1,6 +1,7 @@
 ---
 - name: ensure registry.runnable in /etc/hosts


wait why do you still have this? we can remove this now right

thejsj · 2017-03-25T01:20:42Z

Missing before merge:

Deploy docker-listener with changes to make sure it works

…e-shared-dock-ami * 'master' of github.com:CodeNow/devops-scripts: Sorted Updated vars for deploy

thejsj · 2017-04-06T23:43:16Z

ansible/roles/dock-images/tasks/main.yml

  become: true
  command: docker pull {{ item }}
  with_items:
-    - "registry.runnable.com/runnable/image-builder:v4.3.0"


@anandkumarpatel and I talked about this. We DO want to bake this image into the AMI in order to make first builds fast, but we have to push image-builder into docker hub as a public image.

image-builder twice Had a problem with starting up Docker and having it use the config with the keys vs it not using it, so change the order for this in order for the first part to use insecure docker for pulling images and then switch to docker with certs + keys.

thejsj added 9 commits March 12, 2017 17:21

Remove loggly

3b1b007

Replace local registry with docker hub

392f95e

Make sure user is sudo

96d9044

Push images to docker hub. Not local registry

ffd27ad

Remove registry.runnable.com

be570dd

Dock should not be bound to any other enviroment. Remove commands tha…

eeca2c8

…t require variables tied to environment

Remove docks-psad

e4db52b

Re add roles. Must be run with skip-tags in order to work

55f3ff6

Change node service config to tag. Skip tags if running dock

a7b1a87

thejsj added 4 commits March 12, 2017 18:03

Remove unused tag

e5f2fcb

Add primary functions for init

c41a0b5

Add vault tokens

425548c

Add tokens

8df4393

Add tag to ensure_registry

982098d

anandkumarpatel requested changes Mar 13, 2017

View reviewed changes

thejsj added 2 commits March 13, 2017 17:17

Fix vault port. Remove amazon-ssm-agent

9400c65

Add line to run dock-init

936f71b

anandkumarpatel requested changes Mar 14, 2017

View reviewed changes

thejsj added 8 commits March 14, 2017 16:28

Add vault port

60bd667

Fix script to run on localhost

3b27d32

Merge branch 'create-shared-dock-ami' of github.com:CodeNow/devops-sc…

e0023b2

…ripts into create-shared-dock-ami * 'create-shared-dock-ami' of github.com:CodeNow/devops-scripts: Remove psad from dock (#613) Added s3 log bucket to ansible

Remove pushing of image

b44c6c6

Make image-builder public

76ec750

Update dock images. Add consul_values tag

c4ebcba

Update krain image

15fee45

thejsj force-pushed the create-shared-dock-ami branch from e6f671f to 15fee45 Compare March 24, 2017 21:44

thejsj added 3 commits March 24, 2017 14:45

Add docker-listener image inspect list

41afba3

Add CA certs. Convert to single template. Overwrite existing files

0cb5bfd

Add more comments

cbd2587

thejsj force-pushed the create-shared-dock-ami branch from ce64411 to cbd2587 Compare March 24, 2017 22:12

thejsj added 3 commits March 24, 2017 15:22

Update vault port

ed929f7

Add user-local to delta

df930de

Fix host addresses. Remove NODE_ENV from krain (default). Change loca…

23dda55

…tion of paths

anandkumarpatel requested changes Mar 25, 2017

View reviewed changes

anandkumarpatel reviewed Mar 25, 2017

View reviewed changes

Change image inspect list. Change author

6d18554

anandkumarpatel approved these changes Mar 25, 2017

View reviewed changes

thejsj added 3 commits April 4, 2017 11:18

Merge branch 'master' of github.com:CodeNow/devops-scripts into creat…

ee5737c

…e-shared-dock-ami * 'master' of github.com:CodeNow/devops-scripts: Sorted Updated vars for deploy

Re-add restart docker

be0ec5f

Change port to consul_api_port

e56c257

thejsj commented Apr 6, 2017

View reviewed changes

thejsj added 2 commits April 6, 2017 21:31

Fix api port and echo in launch config

d2b5e7b

thejsj merged commit bebf0e0 into master Apr 7, 2017

thejsj deleted the create-shared-dock-ami branch April 7, 2017 20:42

		@@ -1,6 +1,7 @@
		---
		- name: ensure registry.runnable in /etc/hosts

Create shared dock ami #611

Create shared dock ami #611

Uh oh!

Conversation

thejsj commented Mar 13, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Bugs/Things that need to be fixed

How to create environment agnostic AMI

Tests

Deployment (post-merge)

TODO for the future

Uh oh!

thejsj commented Mar 13, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thejsj commented Mar 13, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thejsj Mar 25, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thejsj commented Mar 25, 2017

Uh oh!

Choose a reason for hiding this comment

thejsj commented Mar 13, 2017 •

edited

Loading

thejsj commented Mar 13, 2017 •

edited

Loading

thejsj commented Mar 13, 2017 •

edited

Loading

thejsj Mar 25, 2017 •

edited

Loading