[#109030954] Concourse job to create minimal VPC #3
Conversation
Add standard Crown Copyright LICENSE file.
Create a versioned environment specific S3 bucket to store terraform state files. Requires terraform 0.6.4 or higher.
| value = "${aws_vpc.myvpc.id}" | ||
| } | ||
|
|
||
| output "vpc-CIDR" { |
There was a problem hiding this comment.
I think it is better to keep this output name in lowercase, don't you think so? This way we keep consistency.
|
In the tasks for the creation of the bucket and for the VPC, it is written all the task and the script inline in the pipeline. But for the destroy tasks we setup a separated task file .yml and a external script. I think creating a separated What do you think? |
| export fly_target=${FLY_TARGET:-tutorial} | ||
|
|
||
| fly unpause-pipeline --pipeline "${pipeline}" | ||
| curl "${ATC_URL}/pipelines/${pipeline}/jobs/init-bucket/builds" -X POST |
There was a problem hiding this comment.
Do we really want to trigger the pipeline whenever we deploy it? It can be risky/undesirable.
Additionally:
- the task can be triggered several times concurrently if you run the deploy script several time. In this case does not matter but it can be an issue. I am not sure how to prevent that.
- I think the intention of the next command
fly watch -j ${pipeline}/vpcis wait for the current triggered pipeline. But actually it will watch the latest finished job, which does not need to be (and it never is) the one triggered in this execution ofdeploy.sh, but the previous one.
There was a problem hiding this comment.
When you do the curl, you get the build number in the response in a Location header:
$ curl -I http://192.168.100.4:8080/pipelines/cf-deploy/jobs/init-bucket/builds -X POST
HTTP/1.1 302 Found
Location: /pipelines/cf-deploy/jobs/init-bucket/builds/15
Date: Mon, 07 Dec 2015 10:57:57 GMT
Content-Length: 0
Content-Type: text/plain; charset=utf-8
The fly watch command has a -b option to specify the build, but I am not sure how that works:
$ fly watch -h
error: Usage:
fly [OPTIONS] watch [watch-OPTIONS]
Application Options:
-t, --target= Concourse target name or URL (default: http://192.168.100.4:8080)
Help Options:
-h, --help Show this help message
[watch command options]
-j, --job=PIPELINE/JOB Watches builds of the given job
-b, --build= Watches a specific build
There was a problem hiding this comment.
the -b option is the build number, for example:
fly -t "tutorial" watch -j cf-deploy/init-bucket -b '16'
The issue is that we trigger the cf-deploy/init-bucket but watch the next task, cf-deploy/vpc which we don't know the build number (it might be different).
Additionally, we need to wait until the task is triggered by cf-deploy/init-bucket to execute watch against cf-deploy/vpc, if not we will get this error:
$ fly -t "tutorial" watch -j cf-deploy/vpc -b '13'
2015/12/07 11:10:29 build not found
Solutions for this could be:
- Write a script to keep polling the jobs until it finishes.
- Extend the command
flyto support watch all the pipeline, instead of a specific job.
|
As commented in person with @mtekel we might get in the situation that the jobs get triggered several times and run in parallel. To avoid that, @mtekel mention that adding the option I have just tested it and it works. Other option which I did not test is Can you add the |
|
I explicitly did the 2 pipelines differently to show different approaches how it can be done. So that we can see and decide... I'd leave it be for now and see how it develops. Regarding deploy. The idea of deploy.sh is to be equivalent to running make aws - essentially it's a deploy trigger/script you run when you want to deploy. But good point about multiple concurrent triggers. Regarding fly watching jobs - that only works for destroy, where you have just triggered the job. I have not looked at finding ways to wait for the new build and only watch that... |
|
Other feedback is add |
Add terraform files to define deployment VPC, infrastructure subnets (one per zone, by default just in the 1st zone) and office access security group (allows SSH and concourse port 8080 from office CIDRs). Use globals.tf for variables. This file contains variables and values common for all terraform deploys.
TF state bucket doesn't need custom variables. It should instead use common globals (we don't want bucket in different location than VPC).
Starting with ignoring tfstate files.
Do `terraform graph` for all TF parts. This is likely not as strong as `terraform plan`, but it does check syntax, presence of most variables and doesn't require credentials.
Add a link to the repository build status to the readme. Writing of the full README is part of a different story: https://www.pivotaltracker.com/n/projects/1275640/stories/109515840
Add a script to upload a pipeline to concourse. Takes environment name, pipeline name and pipeline yaml config file as parameters. These can be also provided by setting DEPLOY_ENV, PIPELINE and PIPELINE_CONFIG env. variables. You can also set AWS_DEFAULT_REGION to override default `eu-west-1` region and BRANCH environment variable to override default `master` branch setting.
Deploy pipeline currently creates initial S3 state bucket (used for keeping terraform, bosh-init and other states) and the minimalistic VPC (just a single subnet, security group and default routing tables). All the logic (including) scrips is in the deploy.yml file itself. This makes it easier to see what's going on, as the pipeline doesn't depend on any external files to read config from. We also use "dummy" semver resource to trigger downstream builds. This is because currently in concourse only resource changes trigger jobs. Hence to trigger down- stream job you need to change some resource that job depends on. This resource also stores the build number in the state S3 bucket, which might be useful for debug. There is already a feature request for triggering jobs by other jobs: concourse/concourse#92 concourse/concourse#46
This script uploads deploy pipeline and triggers it. It takes env. name as parameter. You can also set git branch of this repo to use by using BRANCH environment variable.
This script is used by tasks in destroy-cf pipeline. We split this out from the main pipeline and task definition, because the script can be used by multiple different tasks and on it's own. This is also to demonstrate a different and usual pipeline organization. With the destroy pipeline, we include tasks from yml files, which run scripts already present in this repository, removing the need to inline the script in the pipeline or task definition.
This task is used by multiple jobs in the destroy pipeline. The task definition by itself is not complete and this is on purpose. The missing field is params (required to set environment variables for tf-destroy script). This is because there is no good default set of parameters and fly only expands parameters from variables in the pipeline and not task. Hence `params` is defined in the pipeline. Another catch with including a task from file in pipeline is that you can't simply add members/items into yml hash or list. E.g. when I want to use `paas-cf` git repo and some S3 bucket as inputs, I can't simply define `paas-cf` in the task in inputs and the S3 bucket resource in the pipeline. The pipeline hash definition overrides whole hash/list in task. There is no merging. Hence I need to include `paas-cf` again explicitly in the pipe definition. This is to highlight the default/recommended organization of items in concourse. We might want to use spruce to merge these files into one big pipeline definition...
Add a pipeline to destroy environments. It currently consists of 2 jobs - one to destroy VPC which then triggers a job to destroy S3 bucket. We used default/recommended pipeline/task/script distribution with this pipeline. We include taks from file, because this task repeats. Never- theless, in this case, the pipeline needs to contain quite chunky config definitions (params), because they can't be included in the task itself, due to fly only expanding params in the pipeline. In comparison with deploy pipeline, we don't use terraform's ability to pull remote state file, due to bug encountered with `bucket.tfstate`: hashicorp/terraform#4154 The remote state file worked with VPC, but we wanted to unify both tasks on same code to demonstrate default concourse pipeline file organization. We also use fake/dummy `trigger-bucket-destroy` resource here to trigger bucket destroy job after vpc destroy job. The terraform bucket destroy job currently doesnt't work, due to bugs: hashicorp/terraform#3981 and missing functionality - the old file versions are actually not removed. @combor has risen PR to add this functionality here: hashicorp/terraform#4168
This script uploads/updates destroy pipeline and then prompts user if they want to continue with triggering the destroy job, which would run the pipeline, which destroys the deployment. Parameters: * environment name * BRANCH as environment variable * AWS_DEFAULT_REGION as environment variable * SKIP_CONFIRM as environment variable. Set this to skip delete confirm dialog.
Make all jobs of deploy and destroy serial, so that we can't have multiple instances of the jobs running at the same state. This is important, because terraform doesn't expect this and we could end up with inconsistent state file.
mtekel
force-pushed
the
109030954-initial-vpc
branch
from
265dca0
to
09dfa05
Compare
Fly outputs state of the last successful job or the job in progress. When you deploy it is quite likely that you run a deploy job before, hence you get output of the last job instead of the build you're starting just now. Add a note that user should likely wait a bit before executing the fly command themselves.
To make sure you really watch the latest build.
The actions in these scripts depend on successful outcome of previous actions. We should therefore fail early.
keymon
force-pushed
the
109030954-initial-vpc
branch
from
d648b76
to
c43aa21
Compare
|
LGTM, merging |
[#109030954] Concourse job to create minimal VPC
Add support for Access Cert and Access Key parsing in the response
…_cloudfoundry [#131375099] Add datadog-for-cloudfoundry to the list of bosh-releases to build
What
Two concourse pipelines to set up initial VPC + S3 bucket and to destroy them on request.
S3 bucket has versioning enabled which is requirement for Concourse S3 resource.
First pipeline is using Terraform to create VPC, networking, security groups and S3 bucket to save state and is triggered by a push to paas-cf repository.
Second pipeline destroys all objects created by first pipeline and has to be executed manually.
Additionally this PR contains:
.gitignoreYou can find more details in commit messages.
How to test
Use your Vagrant box to start concourse server.
Execute
BRANCH=109030954-initial-vpc concourse/scripts/deploy.sh your_env_nameGo to concourse to check whether pipelines are correctly created.
Push a commit to
109030954-initial-vpcand check if cf-deploy was started and finished successfully. Please make sure you don't merge your test commits.Execute
BRANCH=109030954-initial-vpc concourse/scripts/destroy.sh your_env_nameWatch
flyoutput to check if terraform destroys VPC.Please note that due to a bug destroy S3 bucket action will fail. We have created a Terraform PR to fix this.
Who can test
Anyone but @combor, @mtekel , @actionjack , @jimconner