[ws-manager-mk2] Workspace timeouts

[WVerlaek]

Description

Implements workspace timeouts.

Depends on #16181 to mark user activity.

Differences with mk1:

• Check for timeouts using a controller instead of a loop running on a timer:
  • Use a (new) controller to monitor workspaces and add the Timeout condition, instead of a timer running every 15 seconds checking all workspaces in a separate goroutine.
    • The new timeout_controller requeues reconcile events, such that each workspace gets reconciled every 15 seconds.
    • It's implemented as a separate controller than the workspace_controller, to keep the reconcile loop light-weight, as it runs every 15 seconds for each workspace. Both controllers share the same k8s cache provided by the controller manager.
• Handling pod restarts for in-memory cache of workspace's last user activity:
  • Instead of marking all workspaces as Active on ws-manager restarting (to prevent the workspaces from timing out immediately after a restart), keep track of the controller startup time to check for a given workspace if a restart occurred.
  • If a restart happened, handle timeouts differently, by "resetting" the timeout to the time of the restart. See comments in code in the WorkspacePhaseRunning phase for more details.

See https://github.com/gitpod-io/gitpod/blob/main/components/ws-manager/pkg/manager/status.go#L747 to compare against mk1 timeout logic.

Related Issue(s)

Relates to #11416

How to test

Check regular workspace timeout:

• Create a workspace
• Edit the workspace CR and set spec.Timeout to 1m
• Wait 1m
• Observe workspace closes due to a timeout

Check restarting ws-manager-mk2 doesn't close workspaces due to in-memory map of last workspace activity clearing on restart:

• Create a workspace
• Edit the workspace CR and set spec.Timeout to 1m
• Keep the workspace active for >2 minutes (also keep it active during the below steps)
• After 2 minutes, restart ws-manager-mk2
• Check that ws-manager-mk2 doesn't timeout the workspace on restart (due to no longer having a last user activity timestamp for the workspace)

Release Notes

NONE

Documentation

Build Options:

• /werft with-github-actions
  Experimental feature to run the build with GitHub Actions (and not in Werft).
• leeway-no-cache
  leeway-target=components:all
• /werft no-test
  Run Leeway with --dont-test
• /werft publish-to-npm

Preview Environment Options:

• /werft with-local-preview
  If enabled this will build install/preview
• /werft with-preview
• /werft with-large-vm
• /werft with-gce-vm
  If enabled this will create the environment on GCE infra
• /werft with-wsman-mk2
• /werft with-integration-tests=all
  Valid options are all, workspace, webapp, ide, jetbrains, vscode, ssh

[werft-gitpod-dev-com]

started the job as gitpod-build-wv-mk2-timeout-ctrl.1 because the annotations in the pull request description changed
(with .werft/ from main)

[WVerlaek]

/werft run

👍 started the job as gitpod-build-wv-mk2-timeout-ctrl.2
(with .werft/ from main)

[WVerlaek]

/werft run recreate-vm

👍 started the job as gitpod-build-wv-mk2-timeout-ctrl.5
(with .werft/ from main)