[test] TestBackup integration test fails

[csweichel]

Bug description

This is NOT a false positive, the backup doesn't actually succeed:

{"instanceId":"d2810fa4-3f27-4fc8-b2d8-6217d4114061","level":"debug","message":"created temp file for workspace backup upload","serviceContext":{"service":"ws-daemon","version":""},"severity":"DEBUG","size":1536,"time":"2021-08-27T07:27:23Z","userId":"builtin-user-workspace-probe-0000000","workspaceId":"aqua-hamster-kws18kc8"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"open /tmp/wsbkp-d2810fa4-3f27-4fc8-b2d8-6217d4114061-414450780.tar: no such file or directory","instanceId":"d2810fa4-3f27-4fc8-b2d8-6217d4114061","level":"error","message":"op failed","op":"upload layer","serviceContext":{"service":"ws-daemon","version":""},"severity":"ERROR","time":"2021-08-27T07:27:23Z","userId":"builtin-user-workspace-probe-0000000","workspaceId":"aqua-hamster-kws18kc8"}
{"backoff":"1s","instanceId":"d2810fa4-3f27-4fc8-b2d8-6217d4114061","level":"debug","message":"retrying op after backoff","op":"upload layer","serviceContext":{"service":"ws-daemon","version":""},"severity":"DEBUG","time":"2021-08-27T07:27:23Z","userId":"builtin-user-workspace-probe-0000000","workspaceId":"aqua-hamster-kws18kc8"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"open /tmp/wsbkp-d2810fa4-3f27-4fc8-b2d8-6217d4114061-414450780.tar: no such file or directory","instanceId":"d2810fa4-3f27-4fc8-b2d8-6217d4114061","level":"error","message":"op failed","op":"upload layer","serviceContext":{"service":"ws-daemon","version":""},"severity":"ERROR","time":"2021-08-27T07:27:24Z","userId":"builtin-user-workspace-probe-0000000","workspaceId":"aqua-hamster-kws18kc8"}
{"backoff":"2s","instanceId":"d2810fa4-3f27-4fc8-b2d8-6217d4114061","level":"debug","message":"retrying op after backoff","op":"upload layer","serviceContext":{"service":"ws-daemon","version":""},"severity":"DEBUG","time":"2021-08-27T07:27:24Z","userId":"builtin-user-workspace-probe-0000000","workspaceId":"aqua-hamster-kws18kc8"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"open /tmp/wsbkp-d2810fa4-3f27-4fc8-b2d8-6217d4114061-414450780.tar: no such file or directory","instanceId":"d2810fa4-3f27-4fc8-b2d8-6217d4114061","level":"error","message":"op failed","op":"upload layer","serviceContext":{"service":"ws-daemon","version":""},"severity":"ERROR","time":"2021-08-27T07:27:26Z","userId":"builtin-user-workspace-probe-0000000","workspaceId":"aqua-hamster-kws18kc8"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"cannot upload workspace content:\n    github.com/gitpod-io/gitpod/ws-daemon/pkg/content.(*WorkspaceService).uploadWorkspaceContent\n        github.com/gitpod-io/gitpod/ws-daemon/pkg/content/service.go:493\n  - open /tmp/wsbkp-d2810fa4-3f27-4fc8-b2d8-6217d4114061-414450780.tar: no such file or directory","instanceId":"d2810fa4-3f27-4fc8-b2d8-6217d4114061","level":"error","message":"final backup failed","serviceContext":{"service":"ws-daemon","version":""},"severity":"ERROR","time":"2021-08-27T07:27:26Z","userId":"builtin-user-workspace-probe-0000000","workspaceId":"aqua-hamster-kws18kc8"}

Steps to reproduce

In a core-dev context:

cd test
go test -v -run TestBackup ./...

Expected behavior

The test should succeed

Example repository

No response

Anything else?

Requires the changes of #5407 to be merged before the test even gets this far.

[csweichel]

/schedule

[roboquat]

@csweichel: Issue scheduled in the workspace team (WIP: 0)

In response to this:

/schedule

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[csweichel]

/assign

[rl-gitpod]

/assign

[rl-gitpod]

Note that #5407 appears to have fixed this issue, however it does not pass every time (1/10 👋 ) as the second workspace start times out, with the workspace never becoming marked as ready, it stays in WorkspacePhase_INITIALIZING. I'm currently rabbit-holing on why this is the case.