[ws-daemon] Improve download speed of snapshots #13515

aledbf · 2022-10-02T22:27:23Z

Description

Improve the download of prebuild from the remote content storage using multiple connections.

How to test

Add the role new-workspace-cluster to your account
Check the persistent_volume_claim is not checked
Deploy an ephemeral cluster
Open a workspace using a prebuild
Remove the role new-workspace-cluster from your account
Open a workspace using a the same prebuild
Compare the times.

Release Notes

NONE

Werft options:

/werft with-local-preview
If enabled this will build install/preview
/werft with-preview
/werft with-integration-tests=all
Valid options are all, workspace, webapp, ide

werft-gitpod-dev-com · 2022-10-02T22:53:17Z

started the job as gitpod-build-aledbf-snap.6 because the annotations in the pull request description changed
(with .werft/ from main)

aledbf · 2022-10-03T00:18:25Z

/werft run

👍 started the job as gitpod-build-aledbf-snap.7
(with .werft/ from main)

utam0k · 2022-10-03T00:22:15Z

This is an improvement by parallelization, right? I wonder how to impact a node by this change. How about doing loadgen?

components/ws-daemon/pkg/content/initializer.go

aledbf · 2022-10-03T00:28:39Z

How about doing loadgen?

That would require access to prebuilds from the loadgen account and prepare a custom scenario.

utam0k · 2022-10-03T00:29:52Z

components/ws-daemon/pkg/content/initializer.go

+	defer os.Remove(tempFile.Name())
+
+	args := []string{
+		"-x16", "-j12",


Apparently, by default, retries are repeated 5 times. This seems a bit much.
https://aria2.github.io/manual/en/html/aria2c.html#cmdoption-m

utam0k · 2022-10-03T00:31:22Z

That would require access to prebuilds from the loadgen account and prepare a custom scenario.

Shall I put it as a note in the next release of the changelog? Will you have the deployer take a careful look at it?

kylos101 · 2022-10-03T02:02:21Z

That would require access to prebuilds from the loadgen account and prepare a custom scenario.

Shall I put it as a note in the next release of the changelog? Will you have the deployer take a careful look at it?

Hi @utam0k 👋 , luckily @aledbf is the deployer this week (on-call next week). 😉

Note: I think loadgen would require a code change to use a prebuild/snapshot initializer for benchmark. Context, I think it assumes the git initializer.

As an alternative to using loadgen, we could cordon the workspace nodes (but one) of an ephemeral cluster, and test a single node.

For example, start 8 standard Gitpod workspaces as fast as you can, have another person do the same, and observe how the node resources perform, do the workspaces start, etc. How does this perform while backups are happening concurrently? It'll be interesting to see how close we get to the memory limit for ws-daemon.

utam0k · 2022-10-03T02:37:22Z

We are lucky!

Hi @utam0k wave , luckily @aledbf is the deployer this week (on-call next week). wink

Ops, I forgot about it 🙇

Note: I think loadgen would require a code change to use a prebuild/snapshot initializer for benchmark. Context, I think it assumes the git initializer.

Is this a limitation of ws-daemo itself? If I understand correctly, classic content-init should have very few restrictions since it runs on ws-daemon. If I am wrong, please let me know.

It'll be interesting to see how close we get to the memory limit for ws-daemon.

Furisto · 2022-10-03T15:21:58Z

Is this a limitation of ws-daemo itself? If I understand correctly, classic content-init should have very few restrictions since it runs on ws-daemon. If I am wrong, please let me know.

No, it is just a limitation of loadgen at the moment.

utam0k · 2022-10-03T23:57:16Z

No, it is just a limitation of loadgen at the moment.

@Furisto
I guess I don't know enough. May I ask you to elaborate a bit more?

sagor999 · 2022-10-03T23:59:22Z

/hold

sagor999

LGTM
Added hold in case of, but I think discussion is not related to the changes.
@aledbf feel free to unhold when you are ready, so that this change can go into next gen cluster. 🙏

mikenikles · 2022-10-04T15:53:09Z

This was a community-reported bug via #12345. Without a changelog entry, how will I know when this is deployed?

utam0k · 2022-10-04T23:48:40Z

This was a community-reported bug via #12345. Without a changelog entry, how will I know when this is deployed?

If the schedule is as usual, maybe by the end of this week.

roboquat added do-not-merge/work-in-progress do-not-merge/release-note-label-needed size/S labels Oct 2, 2022

[ws-daemon] Improve download speed of snapshots

9bc9652

aledbf force-pushed the aledbf/snap branch from 9c8fad1 to 9bc9652 Compare October 2, 2022 22:28

roboquat added release-note-none and removed do-not-merge/release-note-label-needed labels Oct 3, 2022

utam0k reviewed Oct 3, 2022

View reviewed changes

components/ws-daemon/pkg/content/initializer.go Show resolved Hide resolved

utam0k reviewed Oct 3, 2022

View reviewed changes

aledbf marked this pull request as ready for review October 3, 2022 00:53

aledbf requested a review from a team October 3, 2022 00:53

roboquat removed the do-not-merge/work-in-progress label Oct 3, 2022

github-actions bot added the team: workspace Issue belongs to the Workspace team label Oct 3, 2022

roboquat added the do-not-merge/hold label Oct 3, 2022

sagor999 approved these changes Oct 4, 2022

View reviewed changes

aledbf removed the do-not-merge/hold label Oct 4, 2022

roboquat merged commit 0b1006a into main Oct 4, 2022

roboquat deleted the aledbf/snap branch October 4, 2022 13:57

roboquat added deployed: workspace Workspace team change is running in production deployed Change is completely running in production labels Oct 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ws-daemon] Improve download speed of snapshots #13515

[ws-daemon] Improve download speed of snapshots #13515

aledbf commented Oct 2, 2022 •

edited

Loading

werft-gitpod-dev-com bot commented Oct 2, 2022

aledbf commented Oct 3, 2022 •

edited by werft-gitpod-dev-com bot

Loading

utam0k commented Oct 3, 2022

aledbf commented Oct 3, 2022

utam0k Oct 3, 2022

utam0k commented Oct 3, 2022

kylos101 commented Oct 3, 2022 •

edited

Loading

utam0k commented Oct 3, 2022

Furisto commented Oct 3, 2022

utam0k commented Oct 3, 2022 •

edited

Loading

sagor999 commented Oct 3, 2022

sagor999 left a comment

mikenikles commented Oct 4, 2022

utam0k commented Oct 4, 2022

[ws-daemon] Improve download speed of snapshots #13515

[ws-daemon] Improve download speed of snapshots #13515

Conversation

aledbf commented Oct 2, 2022 • edited Loading

Description

How to test

Release Notes

Werft options:

werft-gitpod-dev-com bot commented Oct 2, 2022

aledbf commented Oct 3, 2022 • edited by werft-gitpod-dev-com bot Loading

utam0k commented Oct 3, 2022

aledbf commented Oct 3, 2022

utam0k Oct 3, 2022

Choose a reason for hiding this comment

utam0k commented Oct 3, 2022

kylos101 commented Oct 3, 2022 • edited Loading

utam0k commented Oct 3, 2022

Furisto commented Oct 3, 2022

utam0k commented Oct 3, 2022 • edited Loading

sagor999 commented Oct 3, 2022

sagor999 left a comment

Choose a reason for hiding this comment

mikenikles commented Oct 4, 2022

utam0k commented Oct 4, 2022

aledbf commented Oct 2, 2022 •

edited

Loading

aledbf commented Oct 3, 2022 •

edited by werft-gitpod-dev-com bot

Loading

kylos101 commented Oct 3, 2022 •

edited

Loading

utam0k commented Oct 3, 2022 •

edited

Loading