BuildKite CI: Deployed buildkite agents can run jobs inside docker containers

[bkase]

Preferably via Docker-out-of-docker (bind-mounting the docker daemon socket) on GKE. As a proof-of-concept, stick a tiny job in a pipeline that runs itself in a different docker container.

Note: There is a discussion on the official buildkite helm charts repo about a quirk of getting this to work on GKE buildkite/charts#54 -- giving a little buffer on time here just case this turns out to be especially annoying.

If necessary this is when we could bake our own buildkite agent image.

Depends on #4802

Epic: #4762

[O1ahmad]

Thinking I get the direction we're heading here:

[buildkite jobs] IN [docker containers] executed by [buildkite agents] as [kubernetes pods] running on GKE:

but yea, just some thoughts on the job execution technique:

• argument against DooD: https://applatix.com/case-docker-docker-kubernetes-part/
• argument for DinD: https://applatix.com/case-docker-docker-kubernetes-part-2/

^^^ only a single opinion here in addition to comments in the linked issue but I'd have to agree with the DinD approach (tl;dr - dind-safe image and sidecar w/ pod-host daemon graph storage mounted), mostly due to simplicity (single container add in buildkite agent pod) but its security and reliability guarantees in a kubernetes context don't hurt either 🛠️

[O1ahmad]

Not sure if we could generate an extra pod container within a helm chart deployment, etc. rendering using the dhall kubernetes/helm components rather than relying on the buildkite helm chart incorporating DinD support or even building our own Docker buildkite agent image...

[O1ahmad]

@bkase @yourbuddyconner, just an update:

• a PR I submitted to buildkite/charts enabling DinD pod sidecar functionality was merged this morning. I've tested with a local minikube setup and seems like it should work for our purposes.
• Buildkite jobs should be able to launch containerized jobs (processes executed in sidecar DinD container) with proper signal propagation and handling by the buildkite-agent container.

[O1ahmad]

Resolved by: https://github.com/CodaProtocol/coda-automation/pull/377