Skip to content
This repository was archived by the owner on Mar 10, 2020. It is now read-only.

feat: upload example works with big files #884

Merged
merged 1 commit into from
Nov 3, 2018

Conversation

lidel
Copy link
Contributor

@lidel lidel commented Nov 1, 2018

Old example used deprecated practice of pre-allocating a Buffer from the file handle.
While it worked fine for small files, with bigger ones introduced artificial limit of ~2GB and caused RangeError: Attempt to allocate Buffer larger than maximum size: 0x7fffffff bytes

People look at examples/upload-file-via-browser to learn "best practices", so I updated it to follow what we learned and do in ipfs-companion, ipfs-webui and ipfs-share-files.

tl;dr

  • This change replaces Buffer with a pull stream, which enables example to work with big files
    (tested with 3.6GB video, took ~30seconds to add to local go-ipfs).

  • While I was at it, I added a demo of a pattern for persisting filenames by wrapping them in unixfs directory (very popular question).

Additional Resources

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
Old example created a Buffer from the file handle which quickly hit
allocation limits with big files.

This change replaces Buffer with a stream, which enables example to work
with big files (tested with 3.6GB video, took ~30seconds with go-ipfs on
localhost).

While I was at it, I added a demo of a pattern for persisting filenames
by wrapping them in unixfs directory.

License: MIT
Signed-off-by: Marcin Rataj <lidel@lidel.org>
Copy link
Contributor

@daviddias daviddias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👏🏽👏🏽👏🏽👏🏽

@daviddias
Copy link
Contributor

CI failed with no-space left on device, as for the rest, looks super cool!

@daviddias daviddias merged commit 62b844f into master Nov 3, 2018
@daviddias daviddias deleted the example/upload-big-file-via-browser branch November 3, 2018 08:07
@ghost ghost removed the in progress label Nov 3, 2018
@rori4
Copy link

rori4 commented Mar 23, 2019

I tested the example and Chromium crashed as the file was too big. The file that I tried to upload was around 400MB
EDIT:
The crash of the browser happens only when I try to use the public infura gateway.
({ host: 'ipfs.infura.io', port: 5001, protocol: 'https' });
On localhost it has no problem. The upload on infura works when I try small files but makes the browser crash when I try to use a big file.

@lidel
Copy link
Contributor Author

lidel commented Mar 24, 2019

@rori4 please open a new issue, and include version of your Chromium. Thank you for reporting this!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants