Epic: Rewrite `ws-manager` as a Kubernetes controller

[atduarte]

Summary

ws-manager is currently a very complex piece of software. This complexity reduces our throughput and increases the chance of introducing errors. That can be mitigated by rewriting it as a Kubernetes controller.

Context

When we built ws-manager more then three years ago, kubebuilder wasn't a thing and writing Kubernetes controllers was hard(er). Also, we didn't know too much about Kubernetes at the time.

Value

• Decreased technical complexity, which means increased throughput, faster team onboarding, and less chance of introducing errors
• Unlocking the possibility of having workspaces that are not pods, but VMs

Acceptance criteria

• Code complexity is significantly reduced
• Code coverage of new implementation is above 70%

Tasks

PoC branch is merged to main

• Introduce compatibility mode in ws-proxy to keep working with ws-manager-mk1
• Rework CRD to match Kubernetes best practice #13335

Basic workspace operation parity

• Introduce foundational controller tests in mk2 #13336
• Remove classic content init in favour of PVC
• Fix CR removal on workspace stop
• Extract WorkspaceAdmission CRD used by ws-proxy #13338

ws-daemon and ws-manager don’t require gRPC anymore

• Store user env vars in secret and reference that secret from CR (OTS removal)
• Replace ws-daemon gRPC call with CR interaction

Prebuild Operation Parity

• Make mk2 understand prebuild pod termination, incl. PVC snapshot creation
• Make ws-daemon upload logs based on CR status
• Support system-failed prebuild restarting
• Ensure correct prebuild status reporting on the gRPC API

mk2 in production for Gitpod.io

• Make mk2's metrics compatible with the current mk1 metrics

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kylos101]

@Furisto @atduarte I updated the status for this to Breakdown, as that's literally what @Furisto and @WVerlaek are doing.

@Furisto , once we're comfortable with the milestone plan, let's plan to move this to in-progress.

[kylos101]

@Furisto @atduarte moved to in-progress, as we're actively working this, and started building a milestone plan with some dates. @Furisto how do you feel about adding additional dates to the milestone plan for tomorrow?