Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support streaming upload of Directories #360

Open
AuHau opened this issue Jun 24, 2021 · 0 comments · May be fixed by #977
Open

Support streaming upload of Directories #360

AuHau opened this issue Jun 24, 2021 · 0 comments · May be fixed by #977
Labels
kind:enhancement A net-new feature or an improvement to an existing feature status:blocked Unable to be worked further until needs are met (see 'need:*' category) type:issue

Comments

@AuHau
Copy link
Contributor

AuHau commented Jun 24, 2021

Currently, it is possible to upload in streaming way file with the uploadFiles() method when Readable is passed. This is possible only in NodeJs at the moment as browsers currently does not support streaming requests. This will hopefully change soon as for example, Chrome has already experimental support for this (https://web.dev/fetch-upload-streaming/).

But... The current implementation of upload of directories relies on the creation of Tar before the upload. This is done all in the memory, so if for example uploading a large directory to Swarm, then everything needs to be loaded into memory, which might lead to Out of Memory errors for Nodejs. It is possible to create Tar in streaming way with tar-stream. We should revisit this to add support for the streaming of directories.

@AuHau AuHau added the kind:enhancement A net-new feature or an improvement to an existing feature label Jun 24, 2021
@AuHau AuHau added the status:blocked Unable to be worked further until needs are met (see 'need:*' category) label Oct 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:enhancement A net-new feature or an improvement to an existing feature status:blocked Unable to be worked further until needs are met (see 'need:*' category) type:issue
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant