[ws-manager] workspaces failed to start

[atduarte]

Bug description

Tried opening workspaces for a side-project and it failed to start at least two times.

Two errors I didn't have time to read showed up in quick succession, and the workspace stopped before it actually started. When I clicked in the "Open Workspace" button that showed up in the "Stopped" page, I got the following error: "cannot initialize workspace: cannot initialize workspace: no backup found".

Steps to reproduce

It was happening for every workspace started, but not anymore.

Workspace affected

atduarte-revispt-960nm6i1khz, atduarte-revispt-jsh4ofkpnyz

Expected behavior

No response

Example repository

No response

Anything else?

No response

[kylos101]

Thanks for the heads up, we initially addressed in generation 44. Therefore, there's still some work to do, scheduled.

[sagor999]

I believe this is due to failure to make prior backup, like when ws-daemon crashes.
To make it easier to debug later on, here is PR: #10491
that makes sure that we will surface if there was an actual error when attempting to download backup.
Once that PR is merged and deployed, I will check logs again to see if there is anything else going on.

[atduarte]

@sagor999 the example workspace ids provided are from fresh starts (of a project with prebuilds) that failed and a few minutes later, trying again, started working.

[sagor999]

atduarte-revispt-960nm6i1khz failed with this error:

cannot pull image: rpc error: code = FailedPrecondition desc = failed to pull and unpack image "reg.ws-eu44xl.gitpod.io:20000/remote/8bef4e73-367a-41b5-94c6-01c1f7d6c879:latest": failed commit on ref "manifest-sha256:844f36297db5efe4e646d7767ef139628f11ca3b0a0a2943b8b7df6226ae3295": "manifest-sha256:844f36297db5efe4e646d7767ef139628f11ca3b0a0a2943b8b7df6226ae3295" failed size validation: 10409 != 10331: failed precondition

which was due to redis\ipfs\cache issues that we saw before.

atduarte-revispt-jsh4ofkpnyz this one did fail with no backup found. Was this WS opened from previous WS that was failed due to that error above? I think in some cases we do not handle the case of failed WS properly and attempt to download backup from it.

[sagor999]

@atduarte moving back into "scheduled" for now. Feel free to close this as well, as I think it might have been resolved since then.

[atduarte]

@sagor999 I'm sorry I missed this.

Was this WS opened from previous WS that was failed due to that error above?

It was a fresh workspace start, that has a prebuild.

I will close it

[kylos101]

We're still seeing this issue, in the saas on gen54, and it's been reported in self-hosted given the May release.

cannot initialize workspace: cannot initialize workspace: no backup found: %!!(MISSING)!(MISSING)!(MISSING)w(<nil>)

Scheduling

[sagor999]

Here is what happens:
https://cloudlogging.app.goo.gl/EGhVdWhrJeqLSnAX6

First workspace failed with:

cannot pull image: rpc error: code = Unknown desc = failed to pull and unpack image "reg.ws-us54.gitpod.io:20000/remote/322c0ace-3122-4798-b180-145f122a254c:latest": failed to copy: httpReadSeeker: failed open: unexpected status code https://reg.ws-us54.gitpod.io:20000/v2/remote/322c0ace-3122-4798-b180-145f122a254c/blobs/sha256:10700734fce6ae2573453f95f5081483a76493753072536a4ffdb46059e50fb0: 500 Internal Server Error - Server message: unknown: unknown error

Then one minute later, second workspace is started (I assume webapp is retrying here):

cannot initialize workspace: cannot initialize workspace: no backup found

This is because of this check (I suspect):

gitpod/components/server/src/workspace/workspace-starter.ts

Lines 274 to 276 in 0945a99

	const hasValidBackup = pastInstances.some(
	(i) => !!i.status && !!i.status.conditions && !i.status.conditions.failed,
	);

That for some reason I think might be saying that it has a validBackup, when in reality it doesn't.
Probably due to this:
https://console.cloud.google.com/logs/query;cursorTimestamp=2022-07-14T10:00:52.387485Z;query=%22322c0ace-3122-4798-b180-145f122a254c%22%0Atimestamp%3D%222022-07-14T10:00:52.322Z%22%0AinsertId%3D%220cy55592tyj86y6s%22;timeRange=2022-07-14T00:32:00.000Z%2F2022-07-14T22:16:00.000Z?project=gitpod-191109
We received an empty "failed" condition overriding an existing one!

Upon further investigation, I believe this is what "clears up" prior failed state:

gitpod/components/ws-manager/pkg/manager/status.go

Lines 349 to 357 in 0945a99

	if isPodBeingDeleted(pod) {
	result.Phase = api.WorkspacePhase_STOPPING
	_, podFailedBeforeBeingStopped := pod.Annotations[workspaceFailedBeforeStoppingAnnotation]
	if !podFailedBeforeBeingStopped {
	// While the pod is being deleted we do not care or want to know about any failure state.
	// If the pod got stopped because it failed we will have sent out a Stopping status with a "failure"
	result.Conditions.Failed = ""
	} else {

PR incoming soon.

[sagor999]

need to redo the fix, as #11419 was reverted due to introduced bug.