[ws-manager-mk2] Maintenance mode

[WVerlaek]

Description

Add maintenance mode for ws-manager-mk2:

• Create ws-manager-mk2-maintenance-mode ConfigMap, which contains a boolean that enables/disables maintenance mode.
  • This ConfigMap is deployed by the installer
• Create a new maintenance reconciler that watches this ConfigMap for changes, and keeps track of whether maintenance mode is enabled
  • The controller manager only watches the maintenance ConfigMap events using a label filter
• The API service returns FailedPrecondition on the following requests when maintenance mode is enabled:
  • StartWorkspace
  • StopWorkspace
  • TakeSnapshot
• The workspace reconciler stops reconciling in maintenance mode, to prevent starting a workspace shutdown and triggering a backup
  • Any events during maintenance mode get requeued after a minute, such that we do quickly reconcile again when maintenance mode ends

Related Issue(s)

Part of #11416

How to test

In this preview env:

• Start a workspace
• Edit the ws-manager-mk2-maintenance-mode ConfigMap and enable maintenance mode
• Try starting a new workspace, stopping the existing workspace, or taking a snapshot, and observe these operations fail
• Restart ws-manager-mk2 and observe it stays in maintenance mode
• Disable maintenance mode again
• Try the above operations which should now succeed

Release Notes

NONE

Documentation

Build Options:

• /werft with-github-actions
  Experimental feature to run the build with GitHub Actions (and not in Werft).
• leeway-no-cache
  leeway-target=components:all
• /werft no-test
  Run Leeway with --dont-test

Publish Options

• /werft publish-to-npm
• /werft publish-to-jb-marketplace

Installer Options

• with-ee-license
• with-dedicated-emulation
• with-ws-manager-mk2
• workspace-feature-flags
  Add desired feature flags to the end of the line above, space separated

Preview Environment Options:

• /werft with-local-preview
  If enabled this will build install/preview
• /werft with-preview
• /werft with-large-vm
• /werft with-gce-vm
  If enabled this will create the environment on GCE infra
• /werft with-integration-tests=all
  Valid options are all, workspace, webapp, ide, jetbrains, vscode, ssh