Uploading large files crashes the browser

[adzuci]

If you try to upload a file that is larger than ~35mb the browser tab will crash about 5 seconds after clicking "Upload".

If we could warn the user about large files or implement a file uploader the user expirience would be better.

# python -c "import IPython; print(IPython.sys_info())"
{'commit_hash': u'fd4ede2',
 'commit_source': 'installation',
 'default_encoding': 'UTF-8',
 'ipython_path': '/usr/local/lib/python2.7/dist-packages/IPython',
 'ipython_version': '4.0.0-dev',
 'os_name': 'posix',
 'platform': 'Linux-3.13.0-49-generic-x86_64-with-Ubuntu-14.04-trusty',
 'sys_executable': '/usr/bin/python',
 'sys_platform': 'linux2',
 'sys_version': '2.7.6 (default, Mar 22 2014, 22:59:56) \n[GCC 4.8.2]'}

[queirozfcom]

Happened to me as well. The RAM used goes way up and then the browser crashes.

[milindaj]

While this issue is being fixed are there any alternatives to uploading larger than 35 MB fies?

[takluyver]

If you're running IPython locally, or it's on a server where you have some other kind of access (ssh, ftp, nfs, Windows file sharing), you can just copy the files you want to the directory where the server is running. If your only access to the server is through IPython, I don't think there's any alternative.

[Carreau]

Bumping to 5.0.

The actually for look that loop the file bytes that encode them as base 64 is crashing.
The alternative is to actually use bits operations which will just push the crash limit.
Or upload binary directly (or in chuncks) which seem more reasonable, but more work.

[julienr]

@Carreau It seems something like resumable.js would help with this. It allows for chuncked and resumable uploads, so it should handles large files fine. This require modifications on the server to store the chunks in a temporary directory and reassemble the files at the end.

[ellisonbg]

Also pinging @blink1073 on this

[blink1073]

We could return a particular error status code if the file is too large, and then use the HTML File API directly as a fallback.

[blink1073]

Well, FileReader, to be more precise.

[blink1073]

Nevermind, I see that you're already using the FileReader in https://github.com/jupyter/notebook/blob/master/notebook/static/tree/js/notebooklist.js. I agree that something else is required if that isn't working, but resumable.js is not available on npm, digging...

[minrk]

Mitigated for 4.1 by #623, which disables upload of large files. Leaving open for 5.0, when we can actually implement chunked upload.

[daniel1124]

We have solved the problem of loading large files by chunking. We also added a progress bar to indicate the upload progress. In our experiment we tried files with a few Gs with no problem.

I would be happy to contribute the code. Please let me know what's the process. Thank you!

[Carreau]

Hi @daniel1124 this was given a shot in #536 currently IIRC the server side does not support receiving the files in chunk so that would need to be improved From the UI side of things, I guess the best path forward may be to directly implement that in jupyterlab which will end up replacing the current notebook, but if you want to try to implment that in normal notebook as well that would be appreciated.

One would have to look how to do chunk file upload with tornado (likely just peaking at the ContentType header + if statement)

Does that make sens ?

[blink1073]

In my opinion this should be done in the Notebook first. The bulk of the work will be in the server and the low-level services calls on the client. A PR will need to be made against this repository either way. A bootstrap progress bar in a bootstrap modal could be used for the Notebook UI, and we can use the new server API and much of the front-end code for the JupyterLab implementation.

[daniel1124]

@blink1073 yeah that's exactly what we did. We modified notebook-4.3.1, mostly in the notebook/static/tree/js/notebooklist.js file, and we add a largefilemanager.py under notebook/service/contents/.

Do you think I should follow the instructions here to contribute? https://github.com/jupyter/notebook/blob/master/CONTRIBUTING.rst

[daniel1124]

@Carreau @blink1073 the way we make it work is that if it is the first chunk, then we set the write mode to "w"; for following chunks we set the write mode to "a". All my changes are confined under the notebook package.

[blink1073]

That sounds great, @daniel1124, and yes, that is the preferred process document.

[daniel1124]

@Carreau @blink1073 I have finished the code and testing, etc. Right now I'm waiting on our company (BlackRock)'s legal/compliance team to give an approval for pull request. We will be contributing through BlackRock-engineering account https://github.com/blackrock.

This is my first time making pull request. Just to understand the process, does the following sound right:

1. we fork the notebook repo into our own repo
2. make changes and commits
3. submit pull request from our own repo
4. wait to be pulled

[takluyver]

@daniel1124 - that process sounds right. Give or take a possible 3.5 - respond to comments from reviewers ;-)

[blink1073]

Great, looking forward to it @daniel1124!

[Carreau]

Thanks @daniel1124 If you or your company need to chat privately for question clarification feel free to reach to any of us privately or contact the Jupyter steering council.

As for your pull-request, you can't break anything so feel free to send even a partial work, it is always possible to update after the fact.

[daniel1124]

@Carreau @blink1073 @takluyver I have added unit tests and cleaned up the code. It's ready for review now.

[daniel1124]

@Carreau @blink1073 @takluyver just made another commit reflecting all the feedback. It's ready for review.

[mawentao119]

Thanks for all your works , I'm waiting for upload large Files.

[Carreau]

@Carreau @blink1073 @takluyver just made another commit reflecting all the feedback. It's ready for review.

Apologies, I was offline for a week or so and I am currently caching up with things.

We'll have a look, right now the notebook codebase is frozen until we publish 5.0, (we are at rc2, so things should be quick), we'll get that in for 5.1.

[takluyver]

The PR for this (#2162) was actually merged for 5.0, despite what we said here.

[mohammedyunus009]

While this issue is being fixed are there any alternatives to uploading larger than 35 MB fies?

Yes there is , u can use filezilla , U just need to get the ssh keys from your sys admin ...... jupyter added it for more user convenience.

[Deepakchawla]

getting the same issue