This repository was archived by the owner on Mar 10, 2020. It is now read-only.
feat: upload example works with big files #884
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Old example created a Buffer from the file handle which quickly hit allocation limits with big files. This change replaces Buffer with a stream, which enables example to work with big files (tested with 3.6GB video, took ~30seconds with go-ipfs on localhost). While I was at it, I added a demo of a pattern for persisting filenames by wrapping them in unixfs directory. License: MIT Signed-off-by: Marcin Rataj <lidel@lidel.org>
ghost
assigned 
|
CI failed with no-space left on device, as for the rest, looks super cool! |
|
I tested the example and Chromium crashed as the file was too big. The file that I tried to upload was around 400MB |
|
@rori4 please open a new issue, and include version of your Chromium. Thank you for reporting this! |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Old example used deprecated practice of pre-allocating a Buffer from the file handle.
While it worked fine for small files, with bigger ones introduced artificial limit of ~2GB and caused
RangeError: Attempt to allocate Buffer larger than maximum size: 0x7fffffff bytesPeople look at
examples/upload-file-via-browserto learn "best practices", so I updated it to follow what we learned and do in ipfs-companion, ipfs-webui and ipfs-share-files.tl;dr
This change replaces Buffer with a pull stream, which enables example to work with big files
(tested with 3.6GB video, took ~30seconds to add to local go-ipfs).
While I was at it, I added a demo of a pattern for persisting filenames by wrapping them in unixfs directory (very popular question).
Additional Resources