[content-service] Refactor upload to GCS #9836

aledbf · 2022-05-06T17:13:04Z

Description

Remove creation of chunks and CRC from the upload of backups. This is one the sources of CPU/RAM utilization in ws-daemon

How to test

Check tests

cd test && \
go test -timeout 300s \
  -run ^TestObjectUpload$ github.com/gitpod-io/gitpod/content-service/pkg/storage -v

or use docker

go test -timeout 300s \
  -run ^TestObjectUpload$ github.com/gitpod-io/gitpod/content-service/pkg/storage -v -args -with-docker

=== RUN   TestObjectUpload
=== RUN   TestObjectUpload/valid_1M_backup
 - - [06/May/2022:15:55:09 -0400] "POST /storage/v1/b?alt=json&prettyPrint=false&project=fake-project HTTP/1.1" 200 161
 - - [06/May/2022:15:55:09 -0400] "GET /storage/v1/b/gitpod-dev-user-fake-owner/o/workspaces%2Ffake-workspace%2Ffull.tar?alt=json&prettyPrint=false&projection=full HTTP/1.1" 404 59
 - - [06/May/2022:15:55:09 -0400] "POST /upload/storage/v1/b/gitpod-dev-user-fake-owner/o?alt=json&name=workspaces%2Ffake-workspace%2Ffull.tar&prettyPrint=false&projection=full&uploadType=multipart HTTP/1.1" 200 536
 - - [06/May/2022:15:55:09 -0400] "GET /gitpod-dev-user-fake-owner/workspaces/fake-workspace/full.tar HTTP/1.1" 200 1001984
=== RUN   TestObjectUpload/valid_100M_backup
 - - [06/May/2022:15:55:09 -0400] "POST /storage/v1/b?alt=json&prettyPrint=false&project=fake-project HTTP/1.1" 200 161
 - - [06/May/2022:15:55:10 -0400] "GET /storage/v1/b/gitpod-dev-user-fake-owner/o/workspaces%2Ffake-workspace%2Ffull.tar?alt=json&prettyPrint=false&projection=full HTTP/1.1" 404 59
 - - [06/May/2022:15:55:10 -0400] "POST /upload/storage/v1/b/gitpod-dev-user-fake-owner/o?alt=json&name=workspaces%2Ffake-workspace%2Ffull.tar&prettyPrint=false&projection=full&uploadType=resumable HTTP/1.1" 200 337
 - - [06/May/2022:15:55:10 -0400] "POST /upload/resumable/0a59d243db09ba73ba636e0dc4c23eb3 HTTP/1.1" 200 474
 - - [06/May/2022:15:55:10 -0400] "POST /upload/resumable/0a59d243db09ba73ba636e0dc4c23eb3 HTTP/1.1" 200 474
 - - [06/May/2022:15:55:10 -0400] "POST /upload/resumable/0a59d243db09ba73ba636e0dc4c23eb3 HTTP/1.1" 200 474
 - - [06/May/2022:15:55:10 -0400] "POST /upload/resumable/0a59d243db09ba73ba636e0dc4c23eb3 HTTP/1.1" 200 474
 - - [06/May/2022:15:55:10 -0400] "POST /upload/resumable/0a59d243db09ba73ba636e0dc4c23eb3 HTTP/1.1" 200 474
 - - [06/May/2022:15:55:10 -0400] "POST /upload/resumable/0a59d243db09ba73ba636e0dc4c23eb3 HTTP/1.1" 200 538
 - - [06/May/2022:15:55:10 -0400] "GET /gitpod-dev-user-fake-owner/workspaces/fake-workspace/full.tar HTTP/1.1" 200 100001792
--- PASS: TestObjectUpload (0.97s)
    --- PASS: TestObjectUpload/valid_1M_backup (0.02s)
    --- PASS: TestObjectUpload/valid_100M_backup (0.95s)
PASS
ok      github.com/gitpod-io/gitpod/content-service/pkg/storage 0.979s

Release Notes

[content-service] Refactor upload to GCS

aledbf · 2022-05-06T22:49:03Z

❯ go test -timeout 60s -run ^TestObjectUpload$ github.com/gitpod-io/gitpod/content-service/pkg/storage -count=1 -v
=== RUN   TestObjectUpload
=== RUN   TestObjectUpload/valid_1M_backup
 - - [06/May/2022:18:47:27 -0400] "POST /storage/v1/b?alt=json&prettyPrint=false&project=fake-project HTTP/1.1" 200 161
 - - [06/May/2022:18:47:27 -0400] "GET /storage/v1/b/gitpod-dev-user-fake-owner/o/workspaces%2Ffake-workspace%2Ffull.tar?alt=json&prettyPrint=false&projection=full HTTP/1.1" 404 59
 - - [06/May/2022:18:47:27 -0400] "POST /upload/storage/v1/b/gitpod-dev-user-fake-owner/o?alt=json&name=workspaces%2Ffake-workspace%2Ffull.tar&prettyPrint=false&projection=full&uploadType=multipart HTTP/1.1" 200 536
 - - [06/May/2022:18:47:27 -0400] "GET /gitpod-dev-user-fake-owner/workspaces/fake-workspace/full.tar HTTP/1.1" 200 1050112
=== RUN   TestObjectUpload/valid_100M_backup
 - - [06/May/2022:18:47:27 -0400] "POST /storage/v1/b?alt=json&prettyPrint=false&project=fake-project HTTP/1.1" 200 161
 - - [06/May/2022:18:47:29 -0400] "GET /storage/v1/b/gitpod-dev-user-fake-owner/o/workspaces%2Ffake-workspace%2Ffull.tar?alt=json&prettyPrint=false&projection=full HTTP/1.1" 404 59
 - - [06/May/2022:18:47:29 -0400] "POST /upload/storage/v1/b/gitpod-dev-user-fake-owner/o?alt=json&name=workspaces%2Ffake-workspace%2Ffull.tar&prettyPrint=false&projection=full&uploadType=resumable HTTP/1.1" 200 337
 - - [06/May/2022:18:47:29 -0400] "POST /upload/resumable/69e4d23e240f9dcb137c28eb4b9fe2ef HTTP/1.1" 200 474
 - - [06/May/2022:18:47:29 -0400] "POST /upload/resumable/69e4d23e240f9dcb137c28eb4b9fe2ef HTTP/1.1" 200 538
 - - [06/May/2022:18:47:29 -0400] "GET /gitpod-dev-user-fake-owner/workspaces/fake-workspace/full.tar HTTP/1.1" 200 104859136
=== RUN   TestObjectUpload/valid_1GB_backup
 - - [06/May/2022:18:47:30 -0400] "POST /storage/v1/b?alt=json&prettyPrint=false&project=fake-project HTTP/1.1" 200 160
 - - [06/May/2022:18:47:33 -0400] "GET /storage/v1/b/gitpod-dev-user-fake-owner/o/workspaces%2Ffake-workspace%2Ffull.tar?alt=json&prettyPrint=false&projection=full HTTP/1.1" 404 59
 - - [06/May/2022:18:47:33 -0400] "POST /upload/storage/v1/b/gitpod-dev-user-fake-owner/o?alt=json&name=workspaces%2Ffake-workspace%2Ffull.tar&prettyPrint=false&projection=full&uploadType=resumable HTTP/1.1" 200 337
 - - [06/May/2022:18:47:33 -0400] "POST /upload/resumable/4f9700cc631a2df2cae7cf8ad7c097a2 HTTP/1.1" 200 474
 - - [06/May/2022:18:47:33 -0400] "POST /upload/resumable/4f9700cc631a2df2cae7cf8ad7c097a2 HTTP/1.1" 200 474
 - - [06/May/2022:18:47:33 -0400] "POST /upload/resumable/4f9700cc631a2df2cae7cf8ad7c097a2 HTTP/1.1" 200 474
 - - [06/May/2022:18:47:33 -0400] "POST /upload/resumable/4f9700cc631a2df2cae7cf8ad7c097a2 HTTP/1.1" 200 474
 - - [06/May/2022:18:47:34 -0400] "POST /upload/resumable/4f9700cc631a2df2cae7cf8ad7c097a2 HTTP/1.1" 200 474
 - - [06/May/2022:18:47:34 -0400] "POST /upload/resumable/4f9700cc631a2df2cae7cf8ad7c097a2 HTTP/1.1" 200 474
 - - [06/May/2022:18:47:35 -0400] "POST /upload/resumable/4f9700cc631a2df2cae7cf8ad7c097a2 HTTP/1.1" 200 474
 - - [06/May/2022:18:47:35 -0400] "POST /upload/resumable/4f9700cc631a2df2cae7cf8ad7c097a2 HTTP/1.1" 200 474
 - - [06/May/2022:18:47:36 -0400] "POST /upload/resumable/4f9700cc631a2df2cae7cf8ad7c097a2 HTTP/1.1" 200 474
 - - [06/May/2022:18:47:37 -0400] "POST /upload/resumable/4f9700cc631a2df2cae7cf8ad7c097a2 HTTP/1.1" 200 474
 - - [06/May/2022:18:47:38 -0400] "POST /upload/resumable/4f9700cc631a2df2cae7cf8ad7c097a2 HTTP/1.1" 200 474
 - - [06/May/2022:18:47:39 -0400] "POST /upload/resumable/4f9700cc631a2df2cae7cf8ad7c097a2 HTTP/1.1" 200 474
 - - [06/May/2022:18:47:40 -0400] "POST /upload/resumable/4f9700cc631a2df2cae7cf8ad7c097a2 HTTP/1.1" 200 474
 - - [06/May/2022:18:47:41 -0400] "POST /upload/resumable/4f9700cc631a2df2cae7cf8ad7c097a2 HTTP/1.1" 200 474
 - - [06/May/2022:18:47:42 -0400] "POST /upload/resumable/4f9700cc631a2df2cae7cf8ad7c097a2 HTTP/1.1" 200 474
 - - [06/May/2022:18:47:43 -0400] "POST /upload/resumable/4f9700cc631a2df2cae7cf8ad7c097a2 HTTP/1.1" 200 474
 - - [06/May/2022:18:47:44 -0400] "POST /upload/resumable/4f9700cc631a2df2cae7cf8ad7c097a2 HTTP/1.1" 200 539
 - - [06/May/2022:18:47:45 -0400] "GET /gitpod-dev-user-fake-owner/workspaces/fake-workspace/full.tar HTTP/1.1" 200 1073743360
--- PASS: TestObjectUpload (19.70s)
    --- PASS: TestObjectUpload/valid_1M_backup (0.01s)
    --- PASS: TestObjectUpload/valid_100M_backup (2.41s)
    --- PASS: TestObjectUpload/valid_1GB_backup (17.27s)
PASS
ok  	github.com/gitpod-io/gitpod/content-service/pkg/storage	19.733s

aledbf · 2022-05-07T23:38:49Z

/hold

csweichel

Took the liberty to push a commit which simplifies the content upload to GCloud further.

I'm not all that hopeful that this change will actually decrease memory use in practice, because of

If anything, the code that existed prior (albeit a ton more complicated) would give us some control over the amount of buffering we do.

aledbf · 2022-05-09T18:36:54Z

Took the liberty to push a commit which simplifies the content upload to GCloud further.

That was on purpose :)
I want to ensure we run the upload in a dedicated goroutine and remove any doubt of interference with other parts of ws-daemon

aledbf · 2022-05-09T18:41:55Z

If anything, the code that existed prior (albeit a ton more complicated) would give us some control over the amount of buffering we do.

but we are setting a value https://github.com/gitpod-io/gitpod/pull/9836/files#diff-c42db953f8fdf32db52a02ec07b01ac4193206a1fe63ffa86201a31dc7e83bd8R374

csweichel · 2022-05-09T18:52:22Z

That was on purpose :)
I want to ensure we run the upload in a dedicated goroutine and remove any doubt of interference with other parts of ws-daemon

Ooops. Dropped that commit again.

csweichel · 2022-05-09T18:57:45Z

If anything, the code that existed prior (albeit a ton more complicated) would give us some control over the amount of buffering we do.

but we are setting a value https://github.com/gitpod-io/gitpod/pull/9836/files#diff-c42db953f8fdf32db52a02ec07b01ac4193206a1fe63ffa86201a31dc7e83bd8R374

We are indeed - good point. Max memory usage this should produce then is numberOfWorkspaces * 64MiB.

aledbf · 2022-05-09T19:29:22Z

Max memory usage this should produce then is numberOfWorkspaces * 64MiB.

I cannot use more than 200MB after the refactoring (I even tried to upload 10GB)

csweichel

🚀

akosyakov

lgtm

geropl · 2022-05-10T15:08:04Z

@aledbf I bet you consider that, but: What's the trade-off in terms of upload speed?

aledbf · 2022-05-10T15:09:06Z

/werft run

👍 started the job as gitpod-build-aledbf-gcs-upload.18
(with .werft/ from main)

aledbf · 2022-05-10T15:13:51Z

What's the trade-off in terms of upload speed?

Good question. Right now we lose nodes due to the memory utilization. That said, I don't see the current behavior to be too fast.

(last 7hs)

geropl

Code changes LGTM, did not test. 👍

roboquat added do-not-merge/work-in-progress do-not-merge/release-note-label-needed size/XXL labels May 6, 2022

aledbf force-pushed the aledbf/gcs-upload branch from c3f442d to 66a23ca Compare May 6, 2022 19:56

roboquat added release-note and removed do-not-merge/release-note-label-needed labels May 6, 2022

aledbf force-pushed the aledbf/gcs-upload branch 4 times, most recently from c7f149c to c8c1a40 Compare May 7, 2022 23:37

aledbf marked this pull request as ready for review May 7, 2022 23:38

aledbf requested a review from a team May 7, 2022 23:38

aledbf requested a review from Furisto as a code owner May 7, 2022 23:38

aledbf requested review from a team May 7, 2022 23:38

aledbf requested review from csweichel and geropl as code owners May 7, 2022 23:38

roboquat removed the do-not-merge/work-in-progress label May 7, 2022

roboquat added the do-not-merge/hold label May 7, 2022

github-actions bot added team: IDE team: delivery Issue belongs to the self-hosted team team: workspace Issue belongs to the Workspace team labels May 7, 2022

aledbf force-pushed the aledbf/gcs-upload branch from 7434300 to 4f5697d Compare May 8, 2022 14:03

csweichel reviewed May 9, 2022

View reviewed changes

csweichel force-pushed the aledbf/gcs-upload branch from bd91014 to 4f5697d Compare May 9, 2022 18:54

csweichel approved these changes May 10, 2022

View reviewed changes

aledbf removed the do-not-merge/hold label May 10, 2022

akosyakov approved these changes May 10, 2022

View reviewed changes

aledbf force-pushed the aledbf/gcs-upload branch from 4f5697d to 3af1791 Compare May 10, 2022 12:56

mrsimonemms approved these changes May 10, 2022

View reviewed changes

aledbf force-pushed the aledbf/gcs-upload branch from 3af1791 to 4fb08c0 Compare May 10, 2022 13:55

aledbf added 6 commits May 10, 2022 11:35

Refactor GCS test to use a fake server

0fd99c3

Refactor upload to GCS

8f1bfb2

Refactor test to use a container for fake GCS server

15a2ea4

Detect docker before running tests

c7d02d9

Use flags instead of docker auto-detection

c26f36b

Update go modules

19cef8d

aledbf force-pushed the aledbf/gcs-upload branch from 4fb08c0 to 19cef8d Compare May 10, 2022 15:35

geropl approved these changes May 12, 2022

View reviewed changes

Furisto approved these changes May 12, 2022

View reviewed changes

roboquat merged commit bbacba4 into main May 12, 2022

roboquat deleted the aledbf/gcs-upload branch May 12, 2022 11:09

roboquat added the deployed: IDE IDE change is running in production label May 17, 2022

kylos101 mentioned this pull request May 30, 2022

[agent-smith] improve performance #10331

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[content-service] Refactor upload to GCS #9836

[content-service] Refactor upload to GCS #9836

aledbf commented May 6, 2022 •

edited by princerachit

Loading

aledbf commented May 6, 2022 •

edited by werft-gitpod-dev-com bot

Loading

aledbf commented May 7, 2022

csweichel left a comment

aledbf commented May 9, 2022

aledbf commented May 9, 2022

csweichel commented May 9, 2022 •

edited

Loading

csweichel commented May 9, 2022

aledbf commented May 9, 2022 •

edited

Loading

csweichel left a comment

akosyakov left a comment

geropl commented May 10, 2022

aledbf commented May 10, 2022 •

edited by werft-gitpod-dev-com bot

Loading

aledbf commented May 10, 2022

geropl left a comment

[content-service] Refactor upload to GCS #9836

[content-service] Refactor upload to GCS #9836

Conversation

aledbf commented May 6, 2022 • edited by princerachit Loading

Description

How to test

Release Notes

aledbf commented May 6, 2022 • edited by werft-gitpod-dev-com bot Loading

aledbf commented May 7, 2022

csweichel left a comment

Choose a reason for hiding this comment

aledbf commented May 9, 2022

aledbf commented May 9, 2022

csweichel commented May 9, 2022 • edited Loading

csweichel commented May 9, 2022

aledbf commented May 9, 2022 • edited Loading

csweichel left a comment

Choose a reason for hiding this comment

akosyakov left a comment

Choose a reason for hiding this comment

geropl commented May 10, 2022

aledbf commented May 10, 2022 • edited by werft-gitpod-dev-com bot Loading

aledbf commented May 10, 2022

geropl left a comment

Choose a reason for hiding this comment

aledbf commented May 6, 2022 •

edited by princerachit

Loading

aledbf commented May 6, 2022 •

edited by werft-gitpod-dev-com bot

Loading

csweichel commented May 9, 2022 •

edited

Loading

aledbf commented May 9, 2022 •

edited

Loading

aledbf commented May 10, 2022 •

edited by werft-gitpod-dev-com bot

Loading