Make TakeSnapshot more reliable

[geropl]

Description

This fixes #5862 by following this idea: #5862 (comment)

In detail

• we add a state: 'pending' | 'error' | 'available'' field to Snapshot (+ message and availableTime)
• we add a method content-service.WorkspaceSnapshotExists
• we make sure ws-daemon.TakeSnapshot returns early
• we add an API method server.waitForSnapshot
• we add a SnapshotService to server which is responsible for making sure all
• we change the semantics fo server.takeSnapshot to:
  • return as early as we have the upload file URL from ws-daemon, stored that in the DB (with state: 'pending') and return the snapshotId
  • we make sure SnapshotService monitors this DB entry and regularly checks if it's done or timed out, resulting in an update to the DB
  • clients have to take that snapshotId and have to call server.waitForSnapshot to wait for the snapshot to become available (or error in case sth went south). This internally waits for the same Promise SnapshotService is already driving

Regarding deployment

This change has breaking changes to IDE and workspace. To untangle and make the deployment as easy as possible:

• once reviewed (and tested) split this PR into two: IDE and workspace + meta
• server and workspace are fully backwards-compatible to each other; server API is backwards compatible as well. Thus we have to make sure meta is deployed before IDE, the rest does not matter.

Related Issue(s)

Fixes #5862

IDE companion PR: gitpod-io/openvscode-server#169

How to test

• open a dev workspace on this branch: https://gitpod.io/#https://github.com/gitpod-io/gitpod/pull/6144
• prepare to execute kubectl rollout restart deployment proxy
• start a test workspace with this link: https://gpl-5862-snapshot.staging.gitpod-dev.com/#https://github.com/shaal/gitpod-test-failing-snapshot (thx @shaal ! 🙏 )
• F1 -> snapshot -> enter
• execute kubectl rollout restart deployment proxy from the dev workspace (this simulates ws connection loss)
• note how the call fails (shows that it's backwards compatible!)
• note how the snapshot process is not interrupted in the IDE of the test workspace
• ~note that after 2mins the snapshot is successful
• test the snapshot link and note that it works

Release Notes

make "snapshots" more reliable

Documentation

• /werft with-clean-slate-deployment

[geropl]

/werft run

👍 started the job as gitpod-build-gpl-5862-snapshot.5

[codecov]

Codecov Report

Merging #6144 (3534f69) into main (0514edc) will increase coverage by 14.43%.
The diff coverage is 0.00%.

❗ Current head 3534f69 differs from pull request most recent head 9ed6299. Consider uploading reports for the commit 9ed6299 to get more accurate results

@@             Coverage Diff             @@
##             main    #6144       +/-   ##
===========================================
+ Coverage   19.04%   33.48%   +14.43%     
===========================================
  Files           2       23       +21     
  Lines         168     4680     +4512     
===========================================
+ Hits           32     1567     +1535     
- Misses        134     2988     +2854     
- Partials        2      125      +123     

Flag	Coverage Δ
components-local-app-app-linux-amd64	?
components-local-app-app-linux-arm64	?
components-local-app-app-windows-386	?
components-local-app-app-windows-amd64	?
components-local-app-app-windows-arm64	?
components-ws-manager-app	39.55% <0.00%> (?)
installer-app	7.45% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
components/ws-manager/pkg/manager/manager_ee.go	0.00% <0.00%> (ø)
components/local-app/pkg/auth/auth.go
components/local-app/pkg/auth/pkce.go
installer/pkg/components/ws-manager/tlssecret.go	0.00% <0.00%> (ø)
components/ws-manager/pkg/manager/metrics.go	11.11% <0.00%> (ø)
installer/pkg/components/ws-manager/rolebinding.go	0.00% <0.00%> (ø)
components/ws-manager/pkg/manager/probe.go	0.00% <0.00%> (ø)
components/ws-manager/pkg/manager/imagespec.go	0.00% <0.00%> (ø)
components/ws-manager/pkg/manager/status.go	73.77% <0.00%> (ø)
...staller/pkg/components/ws-manager/networkpolicy.go	0.00% <0.00%> (ø)
... and 16 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0514edc...9ed6299. Read the comment docs.

[geropl]

/werft run

👍 started the job as gitpod-build-gpl-5862-snapshot.45

another try---

[geropl]

/werft run

👍 started the job as gitpod-build-gpl-5862-snapshot.47

...and again

[geropl]

/werft run with-clean-slate-deployment

👍 started the job as gitpod-build-gpl-5862-snapshot.48

The best things come in threes. 🤞

[geropl]

/unhold

Ready to merge now

[JanKoehnlein]

/lgtm

[roboquat]

LGTM label has been added.

Git tree hash: e74e73cd5ea3964fe6b81a871af7253bdf9429f6

[gtsiolis]

Tried this for a workspace larger than 3.5GB which used to fail and worked as expected! 🔮

Thanks @geropl! 🏁

[csweichel]

did not test, but API and service changes lgtm

/lgtm

[roboquat]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: csweichel, JanKoehnlein

Associated issue: #5862

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• components/content-service/OWNERS [csweichel]
• components/content-service-api/OWNERS [csweichel]
• components/gitpod-db/OWNERS [JanKoehnlein]
• components/gitpod-protocol/OWNERS [JanKoehnlein]
• components/server/OWNERS [JanKoehnlein]
• components/ws-daemon/OWNERS [csweichel]
• components/ws-daemon-api/OWNERS [csweichel]
• components/ws-manager/OWNERS [csweichel]
• components/ws-manager-api/OWNERS [csweichel]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[shaal]

I can't wait for this feature to be working again 🎉
When do you expect this to be available on gitpod.io ?

[gtsiolis]

Thanks, @shaal! The fix should probably land in production within a week.

FYI, we're trying to get better at deploying to production sooner and more often.
We're aiming at deploying at least twice a week. 💡

[geropl]

@gtsiolis Just to clarify: This PR is just introducing the server/workspace changes. IDE change will land later, I expect next week.

[gtsiolis]

Thanks for clarifying, @geropl!

Is there an open issue for the IDE changes?

Cc @shaal

[geropl]

It's the same issue: #5862

Draft PR is here: gitpod-io/openvscode-server#169