moving krain off the docks

.gitignore

@@ -0,0 +1,11 @@
+node_modules
+*.pem
+pass
+hellorunnable
+dump.rdb
+erl_crash.dump
+npm-debug.log
+ca.srl
+.DS_Store
+ansible/roles/hipache/templates/runnable*
+ansible/certs/*

README.md

@@ -1,4 +1,141 @@
 devops-scripts
 ==============
 
-devops-scripts
+Scripts for managing our deployments.
+
+# How to Deploy at Runnable
+## Setup
+
+Before you can deploy you'll need to install the appropriate tools, scripts, and keys on your local machine.
+To do so, execute the following steps:
+
+1. Install Ansible (the deploy automation tool we use to deploy projects to production)
+http://docs.ansible.com/intro_installation.html
+
+2. Get the latest devops-scripts (the recipes that we use to deploy various projects)
+https://github.com/CodeNow/devops-scripts
+
+3. Change to the devops scripts repo directory and run the following command:
+`ln -s /<local-path-to-devops-scripts>/ssh/config ~/.ssh/config`
+
+4. Obtain the “Keys of Power” from someone who can already deploy (ask Anand if you don’t know). Depending on what you want to deploy you'll receive either `Test-runnable.pem`, `oregon.pem`, or both.
+
+5. Move the “Keys of Power” .pem  files to your `~/.ssh` directory
+
+At this point you should be capable of deploying; keep reading to find out how to actually perform a deploy!
+
+## Deploying Services
+- **IMPORTANT:** always pull latest devopts-scripts (`git pull origin master`)
+- **IMPORTANT:** Before you deploy a new version of any project make sure to determine which version of the project is currently deployed. This way you can quickly revert to the last stable release if something goes wrong after pushing a new version.
+
+### Step 1: Determine the Current Deploy Version
+To determine the latest deploy tag for a project please check the project's repository on
+github and look for the latest release tag (should be in the form `vX.Y.Z`). Once you've located the tag,
+copy it down somewhere that is easily and quickly accessible (you may need to use it quickly if something goes wrong).
+
+### Step 2: Deploy the Project via `ansible-playbook`
+
+- **WARNING:** If you were unable to determine the last deploy tag for a project and cannot revert **STOP**.
+  Ask someone on the team for help before continuing.
+- **IMPORTANT:** All commands should be run from the `devops-script/ansible` directory.
+
+#### Latest Tag
+Build and deploy a service to the latest tag of its repository. This will build
+the docker image needed to run the container on our infrastructure.
+
+##### Command
+```
+ansible-playbook -i ./[inventory_dir] [service-playbook]
+```
+
+##### Arguments
+- `[inventory_dir]` - The environment inventory files (servers and variables). Should be one of the following:
+  - `stage-hosts` - Runnable sandbox staging environment services
+  - `gamma-hosts` - Gamma services (internal use only; production mirror)
+  - `delta-hosts` - Delta services (real production)
+- `[service-playbook]` - The playbook for the service you wish to deploy, ex:
+  - `api.yml` - Deploys both the api and the api-workers services
+  - `shiva.yml` - Deploys the shiva micro-service
+  - `charon.yml` - Deploys a specific version of charon DNS to all docks
+
+#### Branch or Tag
+Build and deploy a service to a specific branch or tag on its repository. This performs a build
+of the docker image needed to run the service on our architecture.
+
+##### Command
+```
+ansible-playbook -i ./[inventory_dir] [service-playbook] -e git_branch=[branch-or-tag]
+```
+
+##### Arguments
+- `[inventory_dir]` - The environment inventory files (servers and variables).
+- `[service-playbook]` - The playbook for the service you wish to deploy.
+- `[branch-or-tag]` - The branch or tag you wish to deploy, ex:
+  - `-e git_branch=v1.9.9` (version tag)
+  - `-e git_branch=my-feature-branch` (branch)
+  - `-e git_branch=3928745892364578623` (specific commit)
+
+
+#### Redeploy Tag or Branch (No Build)
+Redeploy the given tag or branch without building the docker image. This is useful for when a redeploy
+is required, but it is of an already built image (e.g. when a service runs out of memory).
+
+##### Command
+```
+ansible-playbook -i ./[inventory_dir] [service-playbook] -t deploy -e git_branch=[branch-or-tag]
+```
+
+##### Arguments
+- `[inventory_dir]` - The environment inventory files (servers and variables).
+- `[service-playbook]` - The playbook for the service you wish to deploy.
+- `[branch-or-tag]` - The branch or tag you wish to deploy.
+
+##### Rebuild and Deploy Tag or Branch (No Cache)
+Forces a rebuild of a docker image for the given service at the given branch or tag and then deploys the
+newly created image. This is useful when a previously deployed branch has new changes that need to
+be deployed to an environment.
+
+Generally this command is only used with `gamma-hosts/` as it is often used to update code
+being tested in the production mirror.
+
+##### Command
+```
+ansible-playbook -i ./[inventory_dir] [service-playbook] -e git_branch=[branch-or-tag] -e build_args=--no-cache
+```
+
+##### Arguments
+- `[inventory_dir]` - The environment inventory files (servers and variables).
+- `[service-playbook]` - The playbook for the service you wish to deploy.
+- `[branch-or-tag]` - The branch or tag you wish to deploy.
+
+
+## Reverting
+If, for some reason, the new deploy is not operating as expected you can quickly revert by referencing the tag you collected in Step 1.
+Simply run the appropriate deploy command in the previous section with the last release tag and the new deploy will be reverted.
+
+## Deploy Songs
+
+- **IMPORTANT:** Make sure to play the song loud and proud when deploying!
+
+It is the custom at Runnable to play a song to the entire team when deploying. For each of the repositories here are the respective songs:
+
+| Service | Deploy Song Link |
+| ------- | ---------------- |
+| api / api-workers | [Push it - Rick Ross](https://www.youtube.com/watch?v=qk2jeE1LOn8) |
+| charon | [Enter Sandman - Metallica](https://www.youtube.com/watch?v=CD-E-LDc384) |
+| detention | [Unbreakable Kimmy Schmidt](https://youtu.be/CV9xF8CjhJk?t=21s) |
+| docker-listener | [Call Me Maybe - Carly Rae Jepsen](https://www.youtube.com/watch?v=fWNaR-rxAic) |
+| filibuster | [He's a Pirate - Pirates Of The Caribbean](https://www.youtube.com/watch?v=yRh-dzrI4Z4) |
+| khronos | [Time After Time - Cyndi Lauper](https://www.youtube.com/watch?v=VdQY7BusJNU) |
+| krain | [Men at Work - Down Under](https://www.youtube.com/watch?v=XfR9iY5y94s) |
+| link | [Zelda Main Theme Song](https://www.youtube.com/watch?v=cGufy1PAeTU) |
+| mavis | [Fairy Tail theme song](https://www.youtube.com/watch?v=R4UFCTMrV-o) |
+| navi | [Ocarina of Time: Lost Woods The Legend of Zelda](https://www.youtube.com/watch?v=iOGpdGEEcJM) |
+| optimus | [Original Transformers Opening Theme](https://www.youtube.com/watch?v=nLS2N9mHWaw) |
+| runnable-angular | [Push it to the limit - Scarface](https://www.youtube.com/watch?v=9D-QD_HIfjA) |
+| sauron | [Sauron theme song from LOTR](https://www.youtube.com/watch?v=V_rk9VBrXMY) |
+| shiva | [FFXIV Shiva Theme](https://www.youtube.com/watch?v=noJiH8HLZw4) |
+| swarm-deamon | [Pink Floyd - Another Brick In The Wall](https://www.youtube.com/watch?v=5IpYOF4Hi6Q) |
+| swarm-manager | [Eric Prydz VS Pink Floyd - 'Proper Education'](https://www.youtube.com/watch?v=IttkDYE33aU) |
+| vault / vault-values | [Seal - Kiss From A Rose](https://www.youtube.com/watch?v=zP3so2hY4CM) |
+| Full Stack Deploy (`all.yml`) | [The Cleveland Orchestra (George Szell conducting) Ludwig von Beethoven Symphony No. 9 "Chorale (Ode To Joy)" Opus 125 IV.] (https://www.youtube.com/watch?v=4g5770gaais) |

ansible/README.md

@@ -0,0 +1,15 @@
+
+Ansible provides a framework for our administration and deployment. It requires an organization for scripts and variables.  By design it uses SSH to connect to all hosts before it executes the actions. As such it can be run from any machine. All Ansible provided functionality is idempotent and it strongly encourage custom scripts match that standard.
+
+Here is the organization of the files in `devops-scripts/ansible`
+
+* `*-hosts` - Files naming all the servers
+* `*.yml` - The top level ansible actions.  These files describe how a host has vars and roles executed on it. 
+* `/group_vars` - yml files that define variables and values for your ansible scripts. This mostly maps one to one with machine types in AWS. They’re a key value map. 
+* `/library` - Third party libraries and scripts. 
+* `/roles` - A set of folders containing the ansible roles. A role defines the executable actions by ansible.  The center pieces is the `/tasks/main.yml`. It defines name actions and requirements. 
+The role can have several sub folders.
+  * `/handlers` - ??? 
+  * `/defaults` - ???
+  * `/meta` - contains dependencies
+  * `/template` - templates for any files that need to be generate and delivered. 

ansible/ansible.cfg

@@ -0,0 +1,3 @@
+[defaults]
+# Required so `sudo: yes` does not lose the environment variables, which hold the ssh-agent socket
+sudo_flags=-HE

ansible/api-core.yml

@@ -0,0 +1,20 @@
+---
+- hosts: mongodb
+- hosts: neo4j
+- hosts: navi
+- hosts: charon
+- hosts: rabbitmq
+- hosts: redis
+- hosts: swarm-manager
+
+- hosts: api
+  vars_files:
+    - "group_vars/alpha-api-base.yml"
+    - "group_vars/alpha-api.yml"
+  roles:
+  - { role: notify, tags: "notify" }
+  - { role: redis_key, tags: ["setup", "redis_key"] }
+  - { role: builder, tags: "build" }
+  - { role: docker_client }
+  - { role: datadog, tags: "datadog" }
+  - { role: container_start }

ansible/api.yml

@@ -0,0 +1,3 @@
+- include: api-core.yml
+- include: socket-server.yml
+- include: workers.yml

ansible/charon.yml

@@ -0,0 +1,13 @@
+---
+- hosts: redis
+- hosts: consul
+
+- hosts: "{{ dock | default('docks') }}"
+  vars_files:
+    - group_vars/alpha-charon.yml
+  roles:
+  - { role: notify, tags: [notify] }
+  - { role: git_repo }
+  - { role: node_service }
+  - { role: loggly-rotate }
+  - { role: consul_value, tags: [consul_value] }

ansible/consul-services.yml

@@ -0,0 +1,7 @@
+---
+- hosts: rabbitmq
+- hosts: redis
+
+- hosts: consul
+  roles:
+  - { role: consul-services }

ansible/consul-values.yml

@@ -0,0 +1,32 @@
+---
+- hosts: consul
+  vars_files:
+    - "group_vars/alpha-consul.yml"
+  tasks:
+    - name: make sure httplib2 is installed
+      sudo: yes
+      apt: package=python-httplib2 state=present
+
+    - name: put values into consul
+      run_once: true
+      when: write_values is defined
+      uri:
+        method=PUT
+        url=http://{{ ansible_default_ipv4.address }}:8500/v1/kv/{{ item.key }}
+        body="{{ item.value }}"
+      with_items: "{{ consul_seed }}"
+
+    - name: get values from consul
+      run_once: true
+      when: read_values is defined
+      uri:
+        method=GET
+        url=http://{{ ansible_default_ipv4.address }}:8500/v1/kv/{{ item.key }}
+      with_items: "{{ consul_seed }}"
+      register: values
+
+    - name: print values to screen
+      run_once: true
+      when: read_values is defined
+      debug: msg="{{ item.item.key }}" -> "{{ item.json[0].Value | b64decode }}"
+      with_items: "{{ values.results }}"

ansible/consul.yml

@@ -0,0 +1,9 @@
+---
+- hosts: consul
+  serial: 1
+  vars_files:
+    - group_vars/alpha-consul.yml
+  roles:
+  - { role: notify, tags: notify }
+  - { role: database }
+  - { role: container_kill_start }

ansible/datadog.yml

@@ -0,0 +1,6 @@
+---
+- hosts: all
+  roles:
+  - { role: base_ubuntu }
+  - { role: apt_update }
+  - { role: datadog, tags: ["datadog"] }

ansible/delta-hosts/docks.js

@@ -0,0 +1,75 @@
+#!/usr/bin/env node
+
+'use strict';
+
+var aws = require('aws-sdk');
+var ec2 = new aws.EC2({
+  accessKeyId: 'AKIAJ3RCYU6FCULAJP2Q',
+  secretAccessKey: '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',
+  region: 'us-west-2'
+});
+
+var params = {
+  Filters: [
+    // Only search for docks in the cluster security group
+    {
+      Name: 'instance.group-id',
+      Values: ['sg-6cd7fb08']
+    },
+    // Only fetch instances that are tagged as docks
+    {
+      Name: 'tag:role',
+      Values: ['dock']
+    },
+    // Only fetch running instances
+    {
+      Name: 'instance-state-name',
+      Values: ['running']
+    }
+  ]
+};
+
+ec2.describeInstances(params, function (err, data) {
+  if (err) {
+    console.error("An error occurred: ", err);
+    process.exit(1);
+  }
+
+  // Get a set of instances from the describe response
+  var instances = [];
+  data.Reservations.forEach(function (res) {
+    res.Instances.forEach(function (instance) {
+      instances.push(instance);
+    });
+  });
+
+  // Map the instances to their private ip addresses
+  // NOTE This will work locally because of the wilcard ssh proxy in the config
+  var hosts = instances.map(function (instance) {
+    return instance.PrivateIpAddress;
+  });
+
+  var hostVars = {};
+  instances.forEach(function (instance) {
+    for (var i = 0; i < instance.Tags.length; i++) {
+      if (instance.Tags[i].Key === 'org') {
+        hostVars[instance.PrivateIpAddress] = {
+          host_tags: instance.Tags[i].Value + ',build,run'
+        };
+      }
+    }
+  });
+
+  // Output the resulting JSON
+  // NOTE http://docs.ansible.com/ansible/developing_inventory.html
+  console.log(JSON.stringify(
+    {
+      docks: {
+        hosts: hosts
+      },
+     _meta : {
+       hostvars : hostVars
+      }
+    }
+  ));
+});