[WIP] Read binary files in chunks

[blink1073]

Fixes #96.

I verified this works with a 340MB file (that crashed in the previous version), and also still works with .ipynb files.

[blink1073]

Well, almost works, the files are corrupted, I'll try again tomorrow...

[blink1073]

This solves the upload portion, but not the download back to the server portion. We need to support a similar chunked protocol on the XHR end.

[blink1073]

When we save() a file, we could send the size as part of the first call to /api/contents/ PUT, then send the blobs after.

[blink1073]

A similar protocol would be needed for get() to support large file download.

[ellisonbg]

@blink1073 do you think this is stable enough to consider for 4.1?

[ellisonbg]

Verbally "no" so marking as 4.2

[tritemio]

This is an important feature for me. I'm using the upload for an online demo to convert data files to a standard format.

If there is something I can do to help to speed up the inclusion of this feature I'd be glad to help.

[blink1073]

I think the simplest way would be to add an optional chunks_remaining field to the REST API model and use the existing base64 encoding. Then you would call save with each chunk from notebooklist.js. The server would have to be updated to detect the presence of chunks_remaining, and keep a file handle open until chunks_remaining is zero. Thoughts?

[takluyver]

Is it worth looking for existing solutions? Uploading a large file over HTTP isn't that unusual, is it?

[blink1073]

From what I've gathered in searching, the consensus is to use the HTML5 FileReader API using binary chunks. There are uploader libraries that handle the chunking/resuming, but we also need to stream downloads from the server. Either way, we'd have to implement the logic on the server to handle streaming.

[Carreau]

That has gone stale, I'm going to close to keep the active PR queue short, and tag with "Closed PR" so that we can reopen when needed. Thanks a lot @blink1073 for getting that started .