Buildkite should provide Docker-in-Docker functionalities

[christian-roggia]

This ticket is related to the following issues happening when using Buildkite on Kubernetes:

• Does not work when running buildkite agent from docker container buildkite-plugins/docker-buildkite-plugin#61
• Docker plugin results in <Segmentation fault> when run in GKE buildkite-plugins/docker-buildkite-plugin#140
• ERROR: mkdir /buildkite: read-only file system agent#729

The following configuration works successfully on Google Kubernetes Engine:

apiVersion: v1
kind: Pod
metadata:
  name: buildkite-agent 
  namespace: buildkite
spec:
  containers:
    - name: agent
      image: buildkite/agent:3.17.0
      imagePullPolicy: Always
      env: 
      - name: BUILDKITE_AGENT_TOKEN
        valueFrom:
          secretKeyRef:
            key: agent-token
            name: buildkite-agent-agent
      - name: BUILDKITE_AGENT_META_DATA
        value: role=agent
      - name: BUILDKITE_BUILD_PATH
        value: /var/buildkite/builds
      - name: DOCKER_HOST
        value: tcp://localhost:2375
      volumeMounts:
        - name: shared-volume
          mountPath: /var/buildkite
    - name: dind
      image: docker:19.03-dind
      securityContext:
        privileged: true
      env:
        - name: DOCKER_TLS_CERTDIR
          value: ""
      volumeMounts:
        - name: docker-graph-storage
          mountPath: /var/lib/docker
        - name: shared-volume
          mountPath: /var/buildkite
  volumes:
    - name: docker-graph-storage
      emptyDir: {}
    - name: shared-volume
      emptyDir: {}

With the following pipeline:

steps:
  - commands:
      - "docker build ."
    plugins:
      - docker#v3.3.0:
          image: "docker:latest"
          always-pull: true
          volumes:
            - "/var/run/docker.sock:/var/run/docker.sock"

I spent around 16 hours trying to figure out how to work around GKE security limitations and how to make Buildkite work with Docker-out-of-Docker, but I ended up reaching the same conclusion I did for Drone CI: Docker-in-Docker is more suitable for the Kubernetes architecture especially for Google Kubernetes Engine architecture.

The following article also explains really well why Docker-in-Docker (dind) is considered more functional in a Kubernetes environment:
https://applatix.com/case-docker-docker-kubernetes-part-2/

The main highlight from the article is the following:

The Pod will create a container that will run outside of the Pod. By running the container using DooD, you lose out on the following for the spawned container:

• Pod Networking - Cannot access the container using Pod IP.
• Pod Lifecycle - On Pod termination, this container will keep running especially if the container was started with -d flag.
• Pod Cleanup - Graph storage will not be cleanup after pod terminates.
• Scheduling and Resource Utilization - Cpu and Memory requested by Pod, will only be for the Pod and not the container spawned from the Pod. Also, limits on CPU and memory set for the Pod will not be inherited by the spawned container.

Aside from that, as far as I know the following tools are all using dind instead of dood when deployed to a k8s environment:

• Gitlab CI
• Drone CI
• Jenkins CI
• GoCD CI

It also looks like AWS is no longer working with DooD without additional configuration (i.e. Buildkite won't work out-of-the-box on ECS without the --enable-docker-bridge flag):
https://support.cloudbees.com/hc/en-us/articles/360028151031-Docker-outside-of-Docker-no-longer-works-in-EKS

This is also the only configuration I am aware of that fully runs on GKE without breaking down when the docker container is used.

[nightah]

I ran into this same problem because the integration tests for our application relies on Docker networking being accessible.

I ended up creating a custom buildkite agent based on the docker:dind container which also bootstrapped some of our relevant CI tools. You can find this at https://github.com/authelia/buildkite.

[rimusz]

@christian-roggia @nightah would you guys care to make a PR to the chart optionally supporting DinD?

[christian-roggia]

@rimusz I will take over this task, but given the limited amount of free time I currently have it might take a few weeks

[christian-roggia]

Unfortunately, I won't be able to work on the DinD chart on a short term, @nightah @rimusz do you mind taking over from here? The information provided in this issue should be enough for the changes required and @nightah example provides even more accurate information.

[O1ahmad]

Hey all, I just submitted a PR which I believe is pretty minimal and allows a wide-range of sidecar use-cases (e.g. DinD) with a single configuration parameter and template injection.

What's nice is that any feature(s) needed by the "podContainers" (e.g. volumes) can be expressed using the chart's existing functionality and should just work.