kubernetes: build with an initContainer or a Job

[maxbrunet]

Context

Currently, envbuilder runs at the start up of a workspace, exposing elements of the buildtime to the runtime and vice-versa:

• Build secrets (e.g. dockerconfig)
• Environment variables (Do not pass techinal ENVs to user #91)
• Mounts (feat: temporarily move rw mounts during image build #187)
• Privileges (Can't do rootless workspaces with coder and envbuilder in k8s cluster #181)
• Container layers are downloaded in each container rather than on the nodes
• ... (feel free to grow the list)

Proposal 1: initContainer

1. Envbuilder would build the image as an initContainer and push it to a container registry
2. The main container would pull and run the image (todo: validate that the pod can be created without the image existing yet)

This would require to generate/know the image reference ahead of time.

Proposal 2: Kubernetes Job

Entire decoupling of buildtime and runtime:

1. Envbuilder runs as Kubernetes Job to build and push the container image
2. It writes a ConfigMap with the reference of the built image
3. Terraform waits for completion of the Job
4. Terraform reads the ConfigMap with kubernetes_config_map datasource (explicitly depending on the Job creation)
5. The image reference from the ConfigMap is then used to create a Deployment
6. A short ttl_seconds_after_finished would allow clean up of the Job for it to be recreated on the next terraform apply

The ConfigMap could be used to share of information between envbuilder and Terraform (#121), like the volumes defined in the devcontainer.json (#220)

Detail to consider: I believe the Coder server starts streaming the logs from the deployment after the terraform apply has finished, it would need to be able to do it for the Job while the apply is running to expose the build logs to the user.

Is it something that has been thought of/done but not documented yet?

[matifali]

Thank you @maxbrunet. I think these are excellent ideas to improve the devcontainer Kubernetes starter template.

[johnstcn]

@maxbrunet these are all great ideas!