This guide describes how to set up Four Keys with your GitHub or GitLab project. The main steps are:
- Forking or cloning this repository
- Building required images with Cloud Build
- Providing values for required Terraform variables
- Executing Terraform to deploy resources
- Generating sample data (optional)
Alternatively, to deploy Four Keys as a remote Terraform module, see
terraform/modules/fourkeys/README.md
To deploy Four Keys with Terraform, you will first need:
- A Google Cloud project with billing enabled
- The owner role assigned to you on the project
- The Google Cloud CLI and Terraform installed on your local machine. We recommend deploying from Cloud Shell on your Google Cloud project.
-
Set an environment variable indicating your Google Cloud project ID:
export PROJECT_ID="YOUR_PROJECT_ID"
-
Clone the fourkeys git repository and change into the root directory
git clone https://github.com/dora-team/fourkeys.git && cd fourkeys
-
Use Cloud Build to build and push containers to Google Container Registry for the dashboard, event-handler:
gcloud builds submit dashboard --config=dashboard/cloudbuild.yaml --project $PROJECT_ID && \ gcloud builds submit event-handler --config=event-handler/cloudbuild.yaml --project $PROJECT_ID
-
Use Cloud Build to build and push containers to Google Container Registry for the parsers you plan to use. See the
bq-workers
for available options. GitHub for example:gcloud builds submit bq-workers --config=bq-workers/parsers.cloudbuild.yaml --project $PROJECT_ID --substitutions=_SERVICE=github
-
Change your working directory to
terraform/example
and renameterraform.tfvars.example
toterraform.tfvars
cd terraform/example && mv terraform.tfvars.example terraform.tfvars
-
Edit
terraform.tfvars
with values for the required variables. Seevariables.tf
for a list of the variables, along with their descriptions and default values. Values not defined interraform.tfvars
will use default values defined invariables.tf
-
Run the following commands from the
example
directory:terraform init
to inialize Terraform and download the moduleterraform plan
to preview changes.terraform apply
to deploy the resources.
To test your Four Keys deployment, you can generate mock data that simulates events from a GitHub repository.
-
Export your event handler URL an environment variable. This is the webhook URL that will receive events:
export WEBHOOK=`gcloud run services list --project $PROJECT_ID | grep event-handler | awk '{print $4}'`
-
Export your event handler secret to an environment variable. This is the secret used to authenticate events sent to the webhook:
export SECRET=`gcloud secrets versions access 1 --secret=event-handler --project $PROJECT_ID`
-
From the root of the fourkeys project run:
python3 data-generator/generate_data.py --vc_system=github
You can see these events being run through the pipeline:
- The event handler logs show successful requests
- The Pub/Sub topic show messages posted
- The BigQuery GitHub parser show successful requests
-
View the generated data in the
events_raw
table in with bq:bq query --project_id $PROJECT_ID 'SELECT * FROM four_keys.events_raw WHERE source = "githubmock";'
Or query the table directly in BigQuery:
SELECT * FROM four_keys.events_raw WHERE source = 'githubmock';