BuildKite CI: Deploying buildkite agents to a cluster v0

[bkase]

We need to build terraform + helm infrastructure to deploy BuildKite agents to GKE.

Getting just this part to work unlocks more parallelizable work.

Epic: #4762

[O1ahmad]

Seems as though we should be able to leverage the official Buildkite Helm chart and Docker image for now and avoid having to develop our own.

With this in mind, the additional infrastructure code structure should resemble the following:

Code Structure

coda-automation
├── helm
├── scripts
├── services
└── terraform
        ├── infrastructure
        ├── modules
                └── kubernetes
                        └── buildkite-agent
        └── buildkite-clusters
                 └── buildkite-product(product CI/CD)
                 └── (e.g. buildkite-infra, buildkite-data, ...) 

[O1ahmad]

Test Infrastructure

@yourbuddyconner @nholland94 - Considering the technical stack for the proposed Buildkite agent cluster infra consists of:

• Docker (container-engine)
• Kubernetes (provisioner/orchestrator) + Helm (Kubernetes package manager)
• Terraform (IaC)

We should be able to enable a CI/CD pipeline for this repository based on:

• minikube + helm
• kubectl's config use-context functionality
• kgoss - a wrapper for goss that aims to bring the simplicity of testing with goss to containers running in pods in Kubernetes
• Terraform (note: based on setting kube context above and perhaps conditional provisioning based on environment and resource type, may be able to imitate exact workflow across environments -- e.g. terraform plan && terraform apply (+ test validation or monitoring/alerting)
• organization-wide or dedicated infrastructure Buildkite cluster

[yourbuddyconner]

I love the use of goss here, and kgoss is especially neat. Overall the plan LGTM

[O1ahmad]

@yourbuddyconner fyi: testing of infrastructure bits/components using kgoss and by extension minikube is currently blocked by minikube-issue #7420.

tl;dr: within minikube, user override of docker-daemon related environment variables such as DOCKER_HOST=, which is used in the podsidecar DinD setup, is ignored as the variables are reset to defaults meant to target only the host docker daemon (unix:///var/run/docker.sock).

Discussion is ongoing re: resolving this by hopefully not resetting overrides and trusting the developer/operator/user to do the right thing, especially in a test or minikube context.

[O1ahmad]

(re: above minikube and DOCKER_HOST issue) Tracking issue: kubernetes/minikube#8219

[O1ahmad]

FYI: CircleCI resource classes/sizing (re: Buildkite agent cluster construction)

[O1ahmad]

v0 of cluster has been provisioned and is operational.

Note: test infrastructure design and implementation will be moved to the v1 cluster iteration.