Terragrunt v0.0.1, with DynamoDB locking

.gitignore

@@ -1,2 +1,3 @@
 .idea
 vendor
+.terragrunt

README.md

@@ -0,0 +1,198 @@
+# Terragrunt
+
+Terragrunt is a thin wrapper for the [Terraform client](https://www.terraform.io/) that provides a distributed locking
+mechanism which allows multiple people to collaborate on the same Terraform state without overwriting each other's
+changes. Terragrunt currently uses Amazon's [DynamoDB](https://aws.amazon.com/dynamodb/) to acquire and release locks.
+DynamoDB is part of the [AWS free tier](https://aws.amazon.com/dynamodb/pricing/), so if you're already using AWS, this
+locking mechanism _should_ be completely free. Other locking mechanisms may be added in the future.
+
+## Motivation
+
+When you use Terraform to provision infrastructure, it records the state of your infrastructure in [state 
+files](https://www.terraform.io/docs/state/). In order to make changes to your infrastructure, everyone on your
+team needs access to these state files. You could check the files into version control (not a great idea, as the state
+files may contain secrets) or use a supported [remote state 
+backend](https://www.terraform.io/docs/state/remote/index.html) to store the state files in a shared location such as 
+[S3](https://www.terraform.io/docs/state/remote/s3.html), 
+[Consul](https://www.terraform.io/docs/state/remote/consul.html), 
+or [etcd](https://www.terraform.io/docs/state/remote/etcd.html). The problem is that none of these options provide 
+*locking*, so if two team members run `terraform apply` on the same state files at the same time, they may overwrite 
+each other's changes. The official solution to this problem is to use [Hashicorp's 
+Atlas](https://www.hashicorp.com/atlas.html), but that requires using a SaaS platform for all Terraform operations and
+can cost a lot of money.
+
+The goal of Terragrunt is to provide a simple, free locking mechanism that allows multiple people to safely collaborate
+on Terraform state.
+
+## Install
+
+1. Install [Terraform](https://www.terraform.io/).
+1. Install Terragrunt by going to the [Releases Page](https://github.com/gruntwork-io/terragrunt/releases), downloading
+   the binary for your OS, and adding it to your PATH.
+
+## Quick start
+
+Go into the folder with your Terraform templates and create a `.terragrunt` file. This file uses the same
+[HCL](https://github.com/hashicorp/hcl) syntax as Terraform and is used to configure Terragrunt and tell it how to do
+locking. To use DynamoDB for locking (see [Locking using DynamoDB](#locking-using-dynamodb)), `.terragrunt` should
+have the following contents:
+
+```hcl
+lockType = "dynamodb"
+stateFileId = "my-app"
+```
+
+Now everyone on your team can use Terragrunt to run all the standard Terraform commands:
+
+```bash
+terragrunt get
+terragrunt plan
+terragrunt apply
+terragrunt output
+terragrunt destroy
+```
+
+Terragrunt forwards most commands directly to Terraform. However, for the `apply` and `destroy` commands, it will first 
+acquire a locking using [DynamoDB](#locking-using-dynamodb):
+
+```
+terragrunt apply
+[terragrunt] 2016/05/27 00:39:18 Attempting to acquire lock for state file my-app in DynamoDB
+[terragrunt] 2016/05/27 00:39:19 Attempting to create lock item for state file my-app in DynamoDB table terragrunt_locks
+[terragrunt] 2016/05/27 00:39:19 Lock acquired!
+terraform apply
+
+aws_instance.example: Creating...
+  ami:                      "" => "ami-0d729a60"
+  instance_type:            "" => "t2.micro"
+
+[...]
+
+Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
+
+[terragrunt] 2016/05/27 00:39:19 Attempting to release lock for state file my-app in DynamoDB
+[terragrunt] 2016/05/27 00:39:19 Lock released!
+```
+
+## Locking using DynamoDB
+
+Terragrunt can use Amazon's [DynamoDB](https://aws.amazon.com/dynamodb/) to acquire and release locks. DynamoDB supports
+[strongly consistent reads](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.DataConsistency.html)
+as well as [conditional writes](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.SpecifyingConditions.html),
+which are all the primitives we need for a very basic distributed lock system. It's also part of [AWS's free
+tier](https://aws.amazon.com/dynamodb/pricing/), and given the tiny amount of data we are working with and the 
+relatively small number of times per day you're likely to run Terraform, it should be a free option for teams already
+using AWS.
+
+#### DynamoDB locking prerequisites
+
+To use DynamoDB for locking, you must:
+
+1. Already have an AWS account.
+1. Set your AWS credentials in the environment using one of the following options:
+    1. Set your credentials as the environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
+    1. Run `aws configure` and fill in the details it asks for.
+    1. Run Terragrunt on an EC2 instance with an IAM Role.
+1. Your AWS user must have an [IAM 
+   policy](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/access-control-identity-based.html) 
+   granting all DynamoDB actions (`dynamodb:*`) on the table `terragrunt_locks` (see the
+   [DynamoDB locking configuration](#dynamodb-locking-configuration) for how to configure this table name). Here is an 
+   example IAM policy that grants the necessary permissions on the `terragrunt_locks` in region `us-west-2` for
+   an account with account id `1234567890`:
+
+    ```json
+    {
+      "Version": "2012-10-17",
+      "Statement": [{
+        "Sid": "",
+        "Effect": "Allow",
+        "Action": "dynamodb:*",
+        "Resource": "arn:aws:dynamodb:us-west-2:1234567890:table/terragrunt_locks"
+      }]
+    }
+    ```
+
+#### DynamoDB locking configuration
+
+For DynamoDB locking, Terragrunt supports the following settings in `.terragrunt`:
+
+```hcl
+lockType = "dynamodb"
+stateFileId = "my-app"
+
+dynamoLock = {
+  awsRegion = "us-east-1"
+  tableName = "terragrunt_locks"
+  maxLockRetries = 360
+}
+```
+
+* `lockType`: (Required) Must be set to `dynamodb`.
+* `stateFileId`: (Required) A unique id for the state file for these Terraform templates. Many teams have more than
+  one set of templates, and therefore more than one state file, so this setting is used to disambiguate locks for one 
+  state file from another.
+* `awsRegion`: (Optional) The AWS region to use. Default: `us-east-1`.
+* `tableName`: (Optional) The name of the table in DynamoDB to use to store lock information. Default:
+  `terragrunt_locks`.
+* `maxLockRetries`: (Optional) The maximum number of times to retry acquiring a lock. Terragrunt waits 10 seconds
+  between retries. Default: 360 retries (one hour).
+
+#### How DynamoDB locking works
+
+When you run `terragrunt apply` or `terragrunt destroy`, Terragrunt does the following:
+
+1. Create the `terragrunt_locks` if it doesn't already exist.
+1. Try to write an item to the `terragrunt_locks` with `stateFileId` equal to the id specified in your
+   `.terragrunt` file. This item will include useful metadata about the lock, such as who created it (e.g. your 
+   username) and when. 
+1. Note that the write is a conditional write that will fail if an item with the same `stateFileId` already exists.
+    1. If the write succeeds, it means we have a lock!
+    1. If the write does not succeed, it means someone else has a lock. Keep retrying every 30 seconds until we get a 
+       lock.
+1. Run `terraform apply` or `terraform destroy`.
+1. When Terraform is done, delete the item from the `terragrunt_locks` to release the lock.
+
+## Cleaning up old locks
+
+If you shut down Terragrunt (e.g. via `CTRL+C`) before it releases a lock, the lock hangs around forever, and will 
+prevent future changes to your state files. To clean up old locks, you can use the `release-lock` command:
+
+```
+terragrunt release-lock
+Are you sure you want to forcibly remove the lock for stateFileId "my-app"? (y/n):
+```
+
+## Developing terragrunt
+
+#### Running all tests
+
+**Note**: The tests for Terragrunt run against a real AWS account and will add and remove real data from DynamoDB.
+
+To run all the tests:
+
+1. Configure your AWS credentials as explained in the [DynamoDB locking prerequisites](dynamodb-locking-prerequisites)
+   section.
+2. `./_ci/run-tests.sh`
+
+#### Running one test
+
+To run a single test, go into the folder with the test and use the `go test` command. Example:
+
+```bash
+cd config
+go test -v -parallel 128
+```
+
+#### Building
+
+```bash
+export VERSION=0.0.1
+gox -os "darwin linux" -output "bin/${APP_NAME}_{{.OS}}_{{.Arch}}" -ldflags="-X main.VERSION=$VERSION"
+```
+
+## TODO
+
+* Implement best-practices in Terragrunt, such as checking if all changes are committed, calling `terraform get`,
+  calling `terraform configure`, etc.
+* Consider implementing alternative locking mechanisms, such as using Git instead of DynamoDB.
+* Consider embedding the Terraform Go code within Terragrunt instead of calling out to it.

_ci/build-and-push-release-assets.sh

@@ -0,0 +1,182 @@
+#!/bin/bash
+#
+# Build golang binaries for all major operating systems, and push them to the given tagged release in GitHub
+#
+
+function print_usage {
+  echo
+  echo "Usage: build-and-push-release-assets.sh [OPTIONS]"
+  echo
+  echo "Build golang binaries for all major operating systems, and push them to the given tagged release in GitHub."
+  echo
+  echo "Options:"
+  echo
+  echo -e "  --local-src-path\t\tThe path where the golang src code can be found."
+  echo -e "  --local-bin-output-path\tThe path where the binaries should be output to."
+  echo -e "  --github-repo-owner\t\tThe owner of the GitHub repo (e.g. gruntwork-io)"
+  echo -e "  --github-repo-name\t\tThe name of the GitHub repo (e.g. terragrunt)"
+  echo -e "  --git-tag\t\t\tThe git tag for which binaries should be built and pushed. We assume that the code at --local-src-path"
+  echo -e "           \t\t\tcorresponds to the git tag."
+  echo -e "  --app-name\t\t\tWhat to name the binary for this app (e.g. terragrunt)"
+  echo
+  echo "Example:"
+  echo
+  echo "  build-and-push-release-assets.sh \ "
+  echo "     --local-src-path \"/home/ubuntu/src\""
+  echo "     --local-bin-output-path \"/home/ubuntu/src\""
+  echo "     --github-repo-owner \"gruntwork-io\""
+  echo "     --github-repo-name \"vaas\""
+  echo "     --git-tag \"v0.0.1\""
+  echo "     --app-name \"terragrunt\""
+}
+
+# Assert that a given binary is installed on this box
+function assert_is_installed {
+  local readonly name="$1"
+
+  if [[ ! $(command -v ${name}) ]]; then
+    echo "ERROR: The binary '$name' is required by this script but is not installed or in the system's PATH."
+    exit 1
+  fi
+}
+
+# Assert that the given command-line arg is non-empty.
+function assert_not_empty {
+  local readonly arg_name="$1"
+  local readonly arg_value="$2"
+
+  if [[ -z "$arg_value" ]]; then
+    echo "ERROR: The value for '$arg_name' cannot be empty"
+    print_usage
+    exit 1
+  fi
+}
+
+# Build go binaries for all major operating systems
+function build_binaries {
+    local readonly local_src_path="$1"
+    local readonly local_bin_output_path="$2"
+    local readonly app_name="$3"
+
+    # build the binaries
+    cd "$local_src_path"
+    gox -os "darwin linux windows" -arch "386 amd64" -output "$local_bin_output_path/${app_name}_{{.OS}}_{{.Arch}}"
+}
+
+# In order to push assets to a GitHub release, we must find the "github tag id" associated with the git tag
+function get_github_tag_id {
+    local readonly github_oauth_token="$1"
+    local readonly git_tag="$2"
+    local readonly github_repo_owner="$3"
+    local readonly github_repo_name="$4"
+
+    curl --silent --show-error \
+         --header "Authorization: token $github_oauth_token" \
+         --request GET \
+         "https://api.github.com/repos/$github_repo_owner/$github_repo_name/releases" \
+         | jq --raw-output ".[] | select(.tag_name==\"$git_tag\").id"
+}
+
+function push_assets_to_github {
+    local readonly local_bin_output_path="$1"
+    local readonly github_oauth_token="$2"
+    local readonly github_tag_id="$3"
+    local readonly github_repo_owner="$4"
+    local readonly github_repo_name="$5"
+
+    # Note that putting "$local_bin_output_path/*" in quotes makes bash expand the * right away, so we deliberately omit quotes
+    local filepath=""
+    for filepath in $local_bin_output_path/*; do
+        # Given a filepath like /a/b/c.txt, return c.txt
+        local readonly filename=$(echo "$filepath" | rev | cut -d"/" -f1 | rev)
+        curl --header "Authorization: token $github_oauth_token" \
+             --header "Content-Type: application/x-executable" \
+             --data-binary @"$filepath" \
+             --request POST \
+             "https://uploads.github.com/repos/$github_repo_owner/$github_repo_name/releases/$github_tag_id/assets?name=$filename"
+    done;
+}
+
+function assert_env_var_not_empty {
+  local readonly var_name="$1"
+  local readonly var_value="${!var_name}"
+
+  if [[ -z "$var_value" ]]; then
+    echo "ERROR: Required environment $var_name not set."
+    exit 1
+  fi
+}
+
+function build_and_push_release_assets {
+  local local_src_path=""
+  local local_bin_output_path=""
+  local github_repo_owner=""
+  local github_repo_name=""
+  local git_tag=""
+  local app_name=""
+
+  assert_env_var_not_empty "$GITHUB_OAUTH_TOKEN"
+  local readonly github_oauth_token="$GITHUB_OAUTH_TOKEN"
+
+  while [[ $# > 0 ]]; do
+    local key="$1"
+
+    case "$key" in
+      --local-src-path)
+        local_src_path="$2"
+        shift
+        ;;
+      --local-bin-output-path)
+        local_bin_output_path="$2"
+        shift
+        ;;
+      --github-repo-owner)
+        github_repo_owner="$2"
+        shift
+        ;;
+      --github-repo-name)
+        github_repo_name="$2"
+        shift
+        ;;
+      --git-tag)
+        git_tag="$2"
+        shift
+        ;;
+      --app-name)
+        app_name="$2"
+        shift
+        ;;
+      --help)
+        print_usage
+        exit
+        ;;
+      *)
+        echo "ERROR: Unrecognized argument: $key"
+        print_usage
+        exit 1
+        ;;
+    esac
+
+    shift
+  done
+
+  assert_is_installed "jq"
+  assert_is_installed "go"
+  assert_is_installed "gox"
+
+  assert_not_empty "--local-src-path" "$local_src_path"
+  assert_not_empty "--local-bin-output-path" "$local_bin_output_path"
+  assert_not_empty "--github-repo-owner" "$github_repo_owner"
+  assert_not_empty "--github-repo-name" "$github_repo_name"
+  assert_not_empty "--git-tag" "$git_tag"
+  assert_not_empty "--app-name" "$app_name"
+
+  build_binaries "$local_src_path" "$local_bin_output_path" "$app_name"
+
+  local github_tag_id=""
+  github_tag_id=$(get_github_tag_id "$github_oauth_token" "$git_tag" "$github_repo_owner" "$github_repo_name")
+
+  push_assets_to_github "$local_bin_output_path" "$github_oauth_token" "$github_tag_id" "$github_repo_owner" "$github_repo_name"
+}
+
+build_and_push_release_assets "$@"