From e7604d969d9e123c472a62863e8dad88bd07d81b Mon Sep 17 00:00:00 2001 From: Tim Gross Date: Tue, 4 Feb 2020 08:49:14 -0500 Subject: [PATCH] e2e: improve provisioning defaults and documentation This changeset improves the ergonomics of running the Nomad e2e test provisioning process by defaulting to a blank `nomad_sha` in the Terraform configuration. By default, a user will now need to pass in one of the Nomad version flags. But they won't have to manually edit the `provisioning.json` file for the common case of deploying a released version of Nomad, and won't need to put dummy values for `nomad_sha`. Includes general documentation improvements. --- e2e/README.md | 108 ++++++++++++++++++++++++--------- e2e/terraform/README.md | 17 ++---- e2e/terraform/main.tf | 3 +- e2e/terraform/terraform.tfvars | 3 - 4 files changed, 88 insertions(+), 43 deletions(-) diff --git a/e2e/README.md b/e2e/README.md index 628193c658..d9275c6051 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -34,39 +34,32 @@ provisioning configuration file for the e2e framework. terraform output provisioning | jq . > ../provisioning.json ``` -By default the `provisioning.json` will include a `nomad_sha` field -for each node. You can edit this file to change the version of Nomad -you want to deploy. Because each node has its own value, you can -create cluster of mixed versions. The provisioning framework accepts -any of the following options: - -- `nomad_sha`: This is a Nomad binary identified by its full commit SHA that's - stored in a shared s3 bucket that Nomad team developers can access. That - commit SHA can be from any branch that's pushed to remote. (Ex. - `"nomad_sha": "0b6b475e7da77fed25727ea9f01f155a58481b6c"`) -- `nomad_local_binary`: This is a path to a Nomad binary on your own host. - (Ex. `"nomad_local_binary": "/home/me/nomad"`) -- `nomad_version`: This is a version number of Nomad that's been released to - HashiCorp. (Ex. `"nomad_version": "0.10.2"`) - -You can pass the following flags to `go test` to override the values -in `provisioning.json` for all nodes: - -- `-nomad.local_file=string`: provision this specific local binary of Nomad -- `-nomad.sha=string`: provision this specific sha from S3 +By default the `provisioning.json` will not include the Nomad version +that will be deployed to each node. You can pass the following flags +to `go test` to set the version for all nodes: + +- `-nomad.local_file=string`: provision this specific local binary of + Nomad. This is a path to a Nomad binary on your own + host. Ex. `-nomad.local_file=/home/me/nomad` +- `-nomad.sha=string`: provision this specific sha from S3. This is a + Nomad binary identified by its full commit SHA that's stored in a + shared s3 bucket that Nomad team developers can access. That commit + SHA can be from any branch that's pushed to + remote. Ex. `-nomad.sha=0b6b475e7da77fed25727ea9f01f155a58481b6c` - `-nomad.version=string`: provision this version from - [releases.hashicorp.com](https://releases.hashicorp.com/nomad) + [releases.hashicorp.com](https://releases.hashicorp.com/nomad). Ex. `-nomad.version=0.10.2` -Deploy Nomad to the cluster: +Then deploy Nomad to the cluster by passing `-provision.terraform` +without a Nomad version flag: ```sh -# from the ./e2e/terraform directory, set your client environment -$(terraform output environment) - -cd .. -go test -v . -provision.terraform ./provisioning.json -skipTests +go test -v . -nomad.version=0.10.2 -provision.terraform ./provisioning.json -skipTests ``` +Because it can take a little while for the cluster to settle, it's +recommended to run this provisioning step (with `-skipTests`) first, +and then run tests as separate step. + ## Running After completing the provisioning step above, you can set the client @@ -84,6 +77,7 @@ go test -v . If you want to run a specific suite, you can specify the `-suite` flag as shown below. Only the suite with a matching `Framework.TestSuite.Component` will be run, and all others will be skipped. + ```sh go test -v -suite=Consul . ``` @@ -95,3 +89,63 @@ names in the full name of the tests: ```sh go test -v . -run 'TestE2E/Consul/\*consul\.ScriptChecksE2ETest/TestGroup' ``` + +## I Want To... + +### ...SSH Into One Of The Test Machines + +You can use the Terraform output to find the IP address. The keys will +in the `./terraform/keys/` directory. + +```sh +ssh -i keys/nomad-e2e-*.pem ubuntu@${EC2_IP_ADDR} +``` + +### ...Deploy a Cluster of Mixed Nomad Versions + +The `provisioning.json` file output by Terraform has a blank field for +`nomad_sha` for each node of the cluster (server and client). You can +manually edit the file to replace this value with a `nomad_sha`, +`nomad_local_binary`, or `nomad_version` for each node to create a +cluster of mixed versions. The provisioning framework accepts any of +the following options for those fields: + +- `nomad_sha`: This is a Nomad binary identified by its full commit + SHA that's stored in a shared s3 bucket that Nomad team developers + can access. That commit SHA can be from any branch that's pushed to + remote. (Ex. `"nomad_sha": + "0b6b475e7da77fed25727ea9f01f155a58481b6c"`) +- `nomad_local_binary`: This is a path to a Nomad binary on your own + host. (Ex. `"nomad_local_binary": "/home/me/nomad"`) +- `nomad_version`: This is a version number of Nomad that's been + released to HashiCorp. (Ex. `"nomad_version": "0.10.2"`) + +Then deploy Nomad to the cluster by passing `-provision.terraform` +without a Nomad version flag: + +```sh +go test -v . -provision.terraform ./provisioning.json -skipTests +``` + +### ...Deploy Custom Configuration Files + +The `provisioning.json` file includes a `bundles` section for each +node of the cluster (server and client). You can manually edit this +file to add, remove, or replace + +```json +"bundles": [ + { + "destination": "/ops/shared/nomad/base.hcl", + "source": "/home/me/custom.hcl" + } +] +``` + +### ...Deploy More Than 4 Linux Clients + +Right now the framework doesn't support this out-of-the-box because of +the way the provisioning script adds specific client configurations to +each client node (for constraint testing). You'll need to add +additional configuration files to +`./e2e/terraform/shared/nomad/indexed`. diff --git a/e2e/terraform/README.md b/e2e/terraform/README.md index b22f178d5e..86770e575a 100644 --- a/e2e/terraform/README.md +++ b/e2e/terraform/README.md @@ -10,7 +10,7 @@ that can be written to a JSON file used by the e2e framework's provisioning step. You can use Terraform to output the provisioning parameter JSON file the e2e -framework uses by setting the `nomad_sha` variable. +framework uses. ## Setup @@ -19,10 +19,7 @@ and `AWS_SECRET_ACCESS_KEY`) to create the Nomad cluster. Use [envchain](https://github.com/sorah/envchain) to store your AWS credentials. Optionally, edit the `terraform.tfvars` file to change the number of -Linux clients, Windows clients, or the Nomad build. You'll usually -want to have the `nomad_sha` variable set here (or via the -`TF_VAR_nomad_sha` env var) so that `terraform output provisioning` is -populated with the build you want. +Linux clients or Windows clients. ```hcl region = "us-east-1" @@ -30,16 +27,13 @@ instance_type = "t2.medium" server_count = "3" client_count = "4" windows_client_count = "1" - -# alternatively, set this via env var: TF_VAR_nomad_sha -# nomad_sha = "" ``` Run Terraform apply to deploy the infrastructure: ```sh cd e2e/terraform/ -TF_VAR_nomad_sha= envchain nomadaws terraform apply +envchain nomadaws terraform apply ``` ## Outputs @@ -75,10 +69,9 @@ ssh -i keys/nomad-e2e-*.pem Administrator@${EC2_IP_ADDR} ## Teardown -The terraform state file stores all the info, so the `nomad_sha` -doesn't need to be valid during teardown. +The terraform state file stores all the info. ```sh cd e2e/terraform/ -envchain nomadaws TF_VAR_nomad_sha=yyyzzz terraform destroy +envchain nomadaws terraform destroy ``` diff --git a/e2e/terraform/main.tf b/e2e/terraform/main.tf index b43db90e1c..540797b038 100644 --- a/e2e/terraform/main.tf +++ b/e2e/terraform/main.tf @@ -34,7 +34,8 @@ variable "windows_client_count" { } variable "nomad_sha" { - description = "The sha of Nomad to run" + description = "The sha of Nomad to write to provisioning output" + default = "" } provider "aws" { diff --git a/e2e/terraform/terraform.tfvars b/e2e/terraform/terraform.tfvars index 6f4add52af..69e4aea71f 100644 --- a/e2e/terraform/terraform.tfvars +++ b/e2e/terraform/terraform.tfvars @@ -3,6 +3,3 @@ instance_type = "t2.medium" server_count = "3" client_count = "4" windows_client_count = "1" - -# alternatively, set this via env var: TF_VAR_nomad_sha -# nomad_sha = ""