Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Avoid allocating too much memory for the buffer on s3 uploads #34537

Merged
merged 1 commit into from
Oct 12, 2022
Merged

Avoid allocating too much memory for the buffer on s3 uploads #34537

merged 1 commit into from
Oct 12, 2022

Conversation

juliushaertl
Copy link
Member

@juliushaertl juliushaertl commented Oct 11, 2022 •
edited
Loading

Otherwise we try to allocate ~500MB in memory which does fail on larger file uploads with our default memory limit. We only need the put file limit configured anyways.

Steps to reproduce on 24+:

  1. use s3 primary storage
  2. upload a 500mb file through the web ui or clients

Before that change the MOVE request ended in an Allowed Memory Size Exhausted error.

@juliushaertl
Avoid allocating too much memory for the buffer
214a50d 
Signed-off-by: Julius Härtl <jus@bitgrid.net>
@juliushaertl
Copy link
Member Author

Regression from https://github.com/nextcloud/server/pull/28542/files#diff-030e38a9e621019981acf91654c1d8d3a12ab3150dbd9fcc4c280479b8c853dbR147

@juliushaertl juliushaertl requested review from a team, skjnldsv and icewind1991 and removed request for a team October 11, 2022 13:39
@juliushaertl
Copy link
Member Author

/backport to stable25

@juliushaertl
Copy link
Member Author

/backport to stable24

skjnldsv
skjnldsv approved these changes Oct 11, 2022
View reviewed changes
Copy link
Member

@skjnldsv skjnldsv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sense! 👍

@juliushaertl juliushaertl changed the title Avoid allocating too much memory for the buffer Avoid allocating too much memory for the buffer on s3 uploads Oct 11, 2022
PVince81
PVince81 approved these changes Oct 11, 2022
View reviewed changes
Copy link
Member

@PVince81 PVince81 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@skjnldsv skjnldsv added 4. to release Ready to be released and/or waiting for tests to finish and removed 3. to review Waiting for reviews labels Oct 12, 2022
@skjnldsv skjnldsv merged commit d936694 into master Oct 12, 2022
@skjnldsv skjnldsv deleted the bugfix/noid/s3-upload-memory branch October 12, 2022 08:24
This was referenced Oct 12, 2022
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@PVince81 PVince81 PVince81 approved these changes

@skjnldsv skjnldsv skjnldsv approved these changes

@CarlSchwan CarlSchwan Awaiting requested review from CarlSchwan

@icewind1991 icewind1991 Awaiting requested review from icewind1991

Assignees
No one assigned
Labels
4. to release Ready to be released and/or waiting for tests to finish bug feature: object storage regression
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@juliushaertl @PVince81 @skjnldsv