[dashboard, server] Prevent redirect loops that trigger "startWorkspace" in a loop

[geropl]

Ways to solve this:

• 1. level:
  • fix the broken assumption in the dashboard (error handling)
• 2. level:
  • have preventive measures in place in dashboard (e.g., exp. backoff for startWorkspace)
  • have preventive measures in place in server: rate-limit startWorkspace more strictly

Proof that this issue enforced this incident: https://ui.honeycomb.io/gitpod/datasets/gitpod-production/result/xnbGHzzXiP7?tab=traces

[sagor999]

Just want to add that we should try to fix it ASAP as it makes us ddos ourselves.

[csweichel]

Just want to add that we should try to fix it ASAP as it makes us ddos ourselves.

💯 agreed.

@jldec can you make sure this gets high up on the priority list?

[jankeromnes]

FYI, a few links to more context about this problem, in case anyone finds this useful:

• Gitpod stuck in starting a workspace screen infinitely #5401
• Launching manual prebuild via URL keeps reloading the page #6688
• Slack thread (internal)

[geropl]

Here's the startWorkspace traces from Friday: https://ui.honeycomb.io/gitpod/datasets/gitpod-production/result/8xyAV6hi4Jj
It shows that for a single user the same workspace is always re-starting the same workspace.

[geropl]

Reproducable in devstaging by:

• copying the relevant ws-daemon label from the node, e.g. gitpod.io/ws-daemon_ready_ns_staging-gpl-8043-redirect-loop: "true"
• scaling down ws-daemon (e.g. by requiring a non-existing node label)
• manually re-adding said ws-daemon_ready... node label 👍

[geropl]

The plan is to:

• [server, dashboard] Do basic rate limiting on startWorkspace #8073: enable basic rate-limiting (per server instance) and handle re-tries in dashboard
• fix the underlying communication issue between the page served by ws-proxy / workspace cluster and the dashboard / app cluster by adding an parameter (not found) that helps the dashboard disting
• (optional) find persistent means to enforece rate-limiting (looking at recent instance starts in the DB OR connect rate-limiter to the DB)

[jldec]

Removed priority: highest (user impact) label given that basic rate-limiter should limit impact to single-user now.