Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate reducing memory usage of chunk uploading #2291

Open
szokeasaurusrex opened this issue Dec 3, 2024 · 0 comments
Open

Investigate reducing memory usage of chunk uploading #2291

szokeasaurusrex opened this issue Dec 3, 2024 · 0 comments

Comments

@szokeasaurusrex
Copy link
Member

Chunk uploading appears to have a very large memory usage when uploading large files, since we appear to be reading the entire file contents into memory. For instance, the (currently experimental) Proguard chunk uploading feature greatly increases memory usage versus when the experimental chunk uploading feature is turned off. While profiling an upload of a 160 MB Proguard file, I observed memory usage of >300 MB while chunk uploading, vs ~90 MB with chunk uploading disabled.

As this problem likely affects all chunk uploads, not just Proguard files, we should investigate if we can reduce memory usage, e.g. by not reading the entire files into memory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant