add a workflow for apply and destroy terraform

.github/workflows/README.md

@@ -0,0 +1,84 @@
+# workflows
+
+These are the automated workflows we use for ensuring a quality working product.
+
+For more on GitHub Actions: <https://docs.github.com/en/actions/>
+
+For more on workflows: <https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions/>
+
+## Contents
+
+- apply-and-destroy-terraform.yml
+
+ This workflow assumes some pre-requisites have been set-up. See: [Configuration Prerequisites](#Configuration-Prerequisites)
+
+ 1. Checks out the .devcontainer from a private container registry for common tools
+
+ 1. Authenticates against a pre-configured KeyVault that contains
+ - values for authenticating against a storage account
+ - values for deploying terraform
+
+ 1. Pulls known good MLZ and Terraform configuration variables from that storage account
+
+ 1. Applies terraform anew from that configuration (see [build/README.md](../../build/README.md) for how this works)
+
+ 1. Destroys terraform from that configuration (see [build/README.md](../../build/README.md) for how this works)
+
+- validate-terraform.yml
+
+ 1. Checks out the .devcontainer from a private container registry for common tools
+
+ 1. Recursively validates and lints all the terraform referenced at src/core
+
+## Configuration Prerequisites
+
+1. MLZ Setup
+
+ To apply terraform at all, locally, or from this automation, `scripts/mlz_tf_setup.sh` must be run to create the storage accounts to store Terraform state and create the Service Principal with authorization to deploy resources into the configured subscription(s).
+
+ See the root README's [Configure the Terraform Backend](#../..//README.md/#Configure-the-Terraform-Backend) on how to do this.
+
+1. Configuration store
+
+ When applying terraform locally or from this automation, an MLZ Configuration file (commonly mlz_tf_cfg.var) and Terraform-specific variables files (commonly *.tfvars) are required.
+
+ You should end up with a container with these files:
+
+ File Name | Value
+ ------------ | -------------
+ mlz_tf_cfg.var | An MLZ Configuration file that comes from mlz_tf_setup.sh
+ globals.tfvars | Global MLZ terraform values
+ saca-hub.tfvars | SACA Hub MLZ terraform values
+ tier-0.tfvars | Tier 0 MLZ terraform values
+ tier-1.tfvars | Tier 1 MLZ terraform values
+ tier-2.tfvars | Tier 2 MLZ terraform values
+
+ Running this from your local machine, you can provide these files yourself, but, today, for automation these files are stored in an Azure Storage Account and retrieved at workflow execution time. See [build/get_vars.sh](../../build/get_vars.sh) to see how we retrieve
+
+ ```plaintext
+ ./build/get_vars.sh
+
+ # pulls down these files:
+ vars/mlz_tf_cfg.var
+ vars/globals.tfvars
+ vars/saca-hub.tfvars
+ vars/tier-0.tfvars
+ vars/tier-1.tfvars
+ vars/tier-2.tfvars
+ ```
+
+1. Secret store and minimally scoped Service Principal
+
+ See [glennmusa/keyvault-for-actions](https://github.com/glennmusa/keyvault-for-actions) to create a minimally scoped Service Principal to pull sensitive values from an Azure Key Vault.
+
+ Supply that Key Vault the values for:
+
+ Secret Name | Value
+ ------------ | -------------
+ MLZCLIENTID | The Service Principal Authorized to deploy resources into MLZ Terraform Subscriptions
+ MLZCLIENTSECRET | The credential for the Service Principal above
+ STORAGEACCOUNT | The Azure Storage Account for the files in the previous step
+ STORAGECONTAINER | The container contianing the files in the previous step
+ STORAGETOKEN | A token to access the storage account (we used a Container SAS)
+
+ For more on creating a minimally scoped token to access storage see: <https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview/>

.github/workflows/apply-and-destroy-terraform.yml

@@ -0,0 +1,60 @@
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT License.
+
+name: apply-and-destroy-terraform
+on: [workflow_dispatch]
+jobs:
+ apply-and-destroy-terraform:
+ runs-on: ubuntu-latest
+
+ container:
+ image: acrmlzcicd.azurecr.io/missionlzdev
+ credentials:
+ username: ${{ secrets.acr_username }}
+ password: ${{ secrets.acr_password }}
+
+ steps:
+ - uses: actions/checkout@v2
+
+ - uses: azure/login@v1
+ with:
+ creds: ${{ secrets.AZURE_CREDENTIALS }}
+
+ - uses: Azure/get-keyvault-secrets@v1
+ with:
+ keyvault: ${{ secrets.KEY_VAULT_NAME }}
+ secrets: '*'
+
+ - name: get vars
+ run : |
+ cd build
+ ./get_vars.sh
+
+ - name: login
+ run : |
+ cd build
+ ./login_azcli.sh vars/mlz_tf_cfg.var
+
+ - name: apply terraform
+ run : |
+ cd build
+ ./apply_tf.sh \
+ vars/mlz_tf_cfg.var \
+ vars/globals.tfvars \
+ vars/saca-hub.tfvars \
+ vars/tier-0.tfvars \
+ vars/tier-1.tfvars \
+ vars/tier-2.tfvars \
+ n
+
+ - name: destroy terraform
+ run : |
+ cd build
+ ./destroy_tf.sh \
+ vars/mlz_tf_cfg.var \
+ vars/globals.tfvars \
+ vars/saca-hub.tfvars \
+ vars/tier-0.tfvars \
+ vars/tier-1.tfvars \
+ vars/tier-2.tfvars \
+ n

build/README.md

@@ -23,26 +23,62 @@ See the root [README's "Configure the Terraform Backend"](../README.md#Configure
 
 Today, the global.tfvars file and the .tfvars for saca-hub, tier0-2, are well known and stored elsewhere. Reach out to the team if you need them.
 
-Then, to apply and destroy pass those six arguments to the relevant script:
+Then, to apply and destroy pass those files as arguments to the relevant script.
+
+There's an [optional argument to display terraform output](#Optionally-display-Terraform-output).
+
+```shell
+usage() {
+ echo "apply_tf.sh: Automation that calls apply terraform given a MLZ configuration and some tfvars"
+ error_log "usage: apply_tf.sh <mlz config> <globals.tfvars> <saca.tfvars> <tier0.tfvars> <tier1.tfvars> <tier2.tfvars> <display terraform output (y/n)>"
+}
+```
 
 ```shell
-# applies terraform in the repo
+# assuming src/scripts/mlz_tf_setup.sh has been run before...
 ./apply_tf.sh \
- ../src/core/mlz_tf_cfg.var \
- ./path_to_vars/globals.tfvars \
- ./path_to_vars/saca-hub.tfvars \
- ./path_to_vars/tier-0.tfvars \
- ./path_to_vars/tier-1.tfvars \
- ./path_to_vars/tier-2.tfvars
+ ./path-to/mlz_tf_cfg.var \
+ ./path-to/globals.tfvars \
+ ./path-to/saca-hub.tfvars \
+ ./path-to/tier-0.tfvars \
+ ./path-to/tier-1.tfvars \
+ ./path-to/tier-2.tfvars \
+ y
 ```
 
 ```shell
-# destroys terraform in the repo
+# assuming src/scripts/mlz_tf_setup.sh has been run before...
 ./destroy_tf.sh \
- ../src/core/mlz_tf_cfg.var \
- ./path_to_vars/globals.tfvars \
- ./path_to_vars/saca-hub.tfvars \
- ./path_to_vars/tier-0.tfvars \
- ./path_to_vars/tier-1.tfvars \
- ./path_to_vars/tier-2.tfvars
+ ./path-to/mlz_tf_cfg.var \
+ ./path-to/globals.tfvars \
+ ./path-to/saca-hub.tfvars \
+ ./path-to/tier-0.tfvars \
+ ./path-to/tier-1.tfvars \
+ ./path-to/tier-2.tfvars \
+ y
 ```
+
+### Optionally display Terraform output
+
+There's an optional argument at the end to specify whether or not to display terraform's output. Set it to 'y' if you want to see things as they happen.
+
+By default, if you do not set this argument, terraform output will be sent to /dev/null (to support clean logs in a CI/CD environment) and your logs will look like:
+
+```plaintext
+Applying saca-hub (1/5)...
+Finished applying saca-hub!
+Applying tier-0 (1/5)...
+Finished applying tier-0!
+Applying tier-1 (1/5)...
+Finished applying tier-1!
+Applying tier-2 (1/5)...
+Finished applying tier-2!
+```
+
+## Gotchas
+
+There's wonky behavior with how Log Analytics Workspaces and Azure Monitor diagnostic log settings are deleted at the Azure Resource Manager level.
+
+For example, if you deployed your environment with Terraform, then deleted it with Azure CLI or the Portal, you can end up with orphan/ghost resources that will be deleted at some other unknown time.
+
+To ensure you're able to deploy on-top of existing resources over and over again, __use Terraform to apply and destroy your environment.__

build/apply_tf.sh

@@ -17,7 +17,7 @@ error_log() {
 
 usage() {
  echo "apply_tf.sh: Automation that calls apply terraform given a MLZ configuration and some tfvars"
- error_log "usage: apply_tf.sh <mlz config> <globals.tfvars> <saca.tfvars> <tier0.tfvars> <tier1.tfvars> <tier2.tfvars>"
+ error_log "usage: apply_tf.sh <mlz config> <globals.tfvars> <saca.tfvars> <tier0.tfvars> <tier1.tfvars> <tier2.tfvars> <display terraform output (y/n)>"
 }
 
 if [[ "$#" -lt 6 ]]; then
@@ -32,40 +32,91 @@ saca_vars=$3
 tier0_vars=$4
 tier1_vars=$5
 tier2_vars=$6
+display_tf_output=${7:-n}
 
 # reference paths
 core_path=$(realpath ../src/core/)
 scripts_path=$(realpath ../src/scripts/)
 
-# source vars from mlz_config
-. "${mlz_config}"
-
 # apply function
 apply() {
  name=$1
- path=$2
- vars=$3
+ tier_sub=$2
+ path=$3
+ vars=$4
 
  # generate config.vars based on MLZ Config and Terraform module
  . "${scripts_path}/config/generate_vars.sh" \
  "${mlz_config}" \
- "${mlz_config_subid}" \
+ "${tier_sub}" \
  "${name}" \
  "${path}"
 
  # remove any existing terraform initialzation
  rm -rf "${path}/.terraform"
 
- # remove any tfvars and subtitute it
+ # copy input vars to temporary file
+ input_vars=$(realpath "${vars}")
+ temp_vars="temp_vars.tfvars"
+ rm -f "${temp_vars}"
+ touch "${temp_vars}"
+ cp "${input_vars}" "${temp_vars}"
+
+ # remove any configuration tfvars and subtitute it with input vars
  tf_vars="${path}/${name}.tfvars"
- rm -rf "${tf_vars}"
- cp "${vars}" "${tf_vars}"
+ rm -f "${tf_vars}"
+ touch "${tf_vars}"
+ cp "${temp_vars}" "${tf_vars}"
+ rm -f "${temp_vars}"
+
+ # set the target subscription
+ az account set \
+ --subscription "${tier_sub}" \
+ --output none
+
+ # attempt to apply $max_attempts times before giving up waiting between attempts
+ # (race conditions, transient errors etc.)
+ apply_success="false"
+ attempts=1
+ max_attempts=5
+
+ apply_command="${scripts_path}/apply_terraform.sh ${globals} ${path} y"
+ destroy_command="${scripts_path}/destroy_terraform.sh ${globals} ${path} y"
 
- "${scripts_path}/apply_terraform.sh" "${globals}" "${path}" "y"
+ if [[ $display_tf_output == "n" ]]; then
+ apply_command+=" &>/dev/null"
+ destroy_command+=" &>/dev/null"
+ fi
+
+ while [ $apply_success == "false" ]
+ do
+ echo "Applying ${name} (${attempts}/${max_attempts})..."
+
+ if ! eval "$apply_command";
+ then
+ # if we fail, run terraform destroy and try again
+ error_log "Failed to apply ${name} (${attempts}/${max_attempts}). Trying some manual clean-up and Terraform destroy..."
+ eval "$destroy_command"
+
+ ((attempts++))
+
+ if [[ $attempts -gt $max_attempts ]]; then
+ error_log "Failed ${max_attempts} times to apply ${name}. Exiting."
+ exit 1
+ fi
+ else
+ # if we succeed meet the base case
+ apply_success="true"
+ echo "Finished applying ${name}!"
+ fi
+ done
 }
 
-# apply terraform
-apply "saca-hub" "${core_path}/saca-hub" "${saca_vars}"
-apply "tier-0" "${core_path}/tier-0" "${tier0_vars}"
-apply "tier-1" "${core_path}/tier-1" "${tier1_vars}"
-apply "tier-2" "${core_path}/tier-2" "${tier2_vars}"
+# source vars from mlz_config
+. "${mlz_config}"
+
+# call apply()
+apply "saca-hub" "${mlz_saca_subid}" "${core_path}/saca-hub" "${saca_vars}"
+apply "tier-0" "${mlz_tier0_subid}" "${core_path}/tier-0" "${tier0_vars}"
+apply "tier-1" "${mlz_tier1_subid}" "${core_path}/tier-1" "${tier1_vars}"
+apply "tier-2" "${mlz_tier2_subid}" "${core_path}/tier-2" "${tier2_vars}"