Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DAV] Download fails for big files: stream_copy_to_stream() throws an error #1707

Closed
derkostka opened this issue Oct 11, 2016 · 32 comments · Fixed by #1740
Closed

[DAV] Download fails for big files: stream_copy_to_stream() throws an error #1707

derkostka opened this issue Oct 11, 2016 · 32 comments · Fixed by #1740

Comments

@derkostka
Copy link
Contributor

derkostka commented Oct 11, 2016

Steps to reproduce

  1. Try to download a big file on Nextcloud, PHP7.0, 32 Bit arm (> 2 GB)
  2. Check downloaded file and log
  3. File is empty, log shows the error immediately

Expected behaviour

File with size of 9 GB should be downloaded

Actual behaviour

0 Byte file created, no error message but logged error

Server configuration

Operating system:
Ubuntu 16.04

Web server:
Apache

Database:
MySQL

PHP version:
PHP 7

Nextcloud version: (see Nextcloud admin page)
Nextcloud 11.0.1

Updated from an older Nextcloud/ownCloud or fresh install:
Updated from Owncloud 9 -> Nextcloud 10 -> Nexcloud 11 alpha
Where did you install Nextcloud from:
Git
Signing status:
List of activated apps:
The content of config/config.php:
Are you using external storage, if yes which one: local/smb/sftp/...
local
Are you using encryption: yes/no
no
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
no

Client configuration

Browser: Chrome

Operating system: MacOS 10.12

Logs

Nextcloud log (data/nextcloud.log)

Nextcloud log

{"reqId":"He91ouATEMBQ7rkEw65e","remoteAddr":"127.0.0.1","app":"PHP","message":"stream_copy_to_stream() expects parameter 3 to be integer, string given at \/media\/data\/www\/nextcloud\/3rdparty\/sabre\/http\/lib\/Sapi.php#78","level":3,"time":"October 11, 2016 20:58:34","method":"GET","url":"\/remote.php\/webdav\/Media-Fernsehaufnahmen\/arte_HD\/arte_HD_Der_Tag_der_Kr%C3%A4hen20160615_135900.ts","user":"name","version":"9.2.0.4"}
{"reqId":"DKWW6ajns8HKX8pETQS+","remoteAddr":"127.0.0.1","app":"PHP","message":"stream_copy_to_stream() expects parameter 3 to be integer, string given at \/media\/data\/www\/nextcloud\/3rdparty\/sabre\/http\/lib\/Sapi.php#78","level":3,"time":"October 11, 2016 20:58:58","method":"GET","url":"\/remote.php\/webdav\/Media-Fernsehaufnahmen\/arte_HD\/arte_HD_Die_wahre_Macht_des_Vatikan_(2_2)20160207_144900.ts","user":"name","version":"9.2.0.4"}
@derkostka
Copy link
Contributor Author

derkostka commented Oct 11, 2016

Update: According to sabre-io/http#61, the issue is located somewhere else. The file size seems to be invalid (NaN) which leads to that error. However, if i apply that patch, it works - but is not solving the root cause as noted in the comments.

Update2: The file breaks at exact 4 GB (real size is 9,1 GB). So there must be an issue somewhere.

Can anybody please test large file downloads in an environment with 32 bit an php 7 ? thx !

@nickvergessen
Copy link
Member

Well you said you use 32 bits, the max int there is 2147483647.
From min to max, the number is 4294967294, which is exactly 4GB.
So I'd say try to switch to 64 bits if you want to have so huge files.

I guess the problem is, that before your cast to int (the intval from the patch), the size is an invalid string. casting to int seems to use the max int, when the value is too high:
See that the following 3 numeric strings all result in the same integer:
https://3v4l.org/OMMWp

var_dump((int) '9223372036854775807');
var_dump((int) '9223372036854775808');
var_dump((int) '92233720368547758081231');

So not sure if there is anything we can do.

@go2sh
Copy link
Contributor

go2sh commented Oct 12, 2016

There seems to be a general issue with files bigger than 2GB on 32-bit systems. 🙈

@derkostka
Copy link
Contributor Author

derkostka commented Oct 13, 2016

There has been a discussion about that topic, generally, stating that as an alternative either floats may be used (which are essentially 64 bits, even on the 32bit arm platform) or numeric strings are an option:
https://github.com/fruux/sabre-dav/issues/408

var_dump((int) '9223372036854775807');

var_dump((int) '9223372036854775808');

var_dump((float) '92233720368547758081231');

results in:

int(9223372036854775807) int(9223372036854775807) float(9.2233720368548E+22)

... but as far as i understood, floats are not an option for sabre dav, so it possibly won´t change. Nextcloud shall catch the error and throw a warnings prior downloading instead, i guess.

@nickvergessen
Copy link
Member

Can you try, whether #1740 fixes the issue?

@nickvergessen
Copy link
Member

And by the way, you should not run any nextcloud 11 alpha version yet. We didn't make any release for that Oo

@derkostka
Copy link
Contributor Author

No, does not seem to solve it. Download is still 0 Byte. Log:

{"reqId":"cIm0o8yJYIzbld34vhUX","remoteAddr":"127.0.0.1","app":"PHP","message":"stream_copy_to_stream() expects parameter 3 to be integer, string given at /media/data/www/nextcloud/3rdparty/sabre/http/lib/Sapi.php#78","level":3,"time":"October 15, 2016 11:24:15","method":"GET","url":"/remote.php/webdav/Media-Fernsehaufnahmen/arte_HD/arte_HD_Der_Tag_der_Kr%C3%A4hen20160615_135900.ts","user":"me","version":"9.2.0.4"}

-> I will investigate and test further by the end of tomorrow.

@derkostka
Copy link
Contributor Author

derkostka commented Oct 19, 2016

The proposed fix did unfortunately not help. So is there really no chance to download files > 4 GB in an 32 Bit Environment ? I am not into the internals of NC, but i think a download is "streamed" in some kind of chunks, so Filesize should not matter for the download itself, correct ? Is it needed to know the actual filesize (as variable) before downloading ?

@nickvergessen
Copy link
Member

Well that seems to exactly be the problem. Before streaming the size is given. But in your case the size is not a number, but a string (because its bigger than the max int) and therefor the error occures.

The problem is, that the log is not really helpful. Let me check if I can find out where this comes from and whether you can add some debug calls for us, that help us finding the real issue.

@nickvergessen
Copy link
Member

Can you make the following changes, try an download and then revert the changes again (otherwise your log will become very huge quickly), and then post the log here?

File 3rdparty/sabre/dav/lib/DAV/CorePlugin.php
find (line 795):

$propFind->handle('{DAV:}getcontentlength', [$node, 'getSize']);

before add:

\OC::$server->getLogger()->error('Debugging propFind(): ' . json_encode($node->getSize()));

File apps/dav/lib/Connector/Sabre/Node.php
find (line 201):

return $this->info->getSize();

before add:

\OC::$server->getLogger()->error('Debugging getSize(): ' . json_encode($this->info->getSize()));

File lib/private/Files/FileInfo.php
find:

return isset($this->data['size']) ? $this->data['size'] : 0;

before add:

\OC::$server->getLogger()->error('Debugging FileInfo::getSize(): ' . json_encode($this->data['size']));

@derkostka
Copy link
Contributor Author

Here are the Results:

File 1, exact Filesize is: 9.085.604.216 Byte
"message":"Debugging propFind(): 9085604216"
"message":"Debugging FileInfo::getSize(): 9085604216"
"message":"Debugging getSize(): 9085604216"
"message":"Debugging FileInfo::getSize(): 9085604216"
--> is correct !

FIle 2, exact Filesize is: 9.585.365.368 Byte:
"message":"Debugging propFind(): 9585365368"
"message":"Debugging FileInfo::getSize(): 9585365368"
"message":"Debugging getSize(): 9585365368"
"message":"Debugging FileInfo::getSize(): 9585365368"
--> is correct !

@nickvergessen
Copy link
Member

Can you try whether https://github.com/nextcloud/server/pull/1902/files helps?

@LukasReschke
Copy link
Member

Also #1890

@derkostka
Copy link
Contributor Author

derkostka commented Oct 25, 2016

Sorry, neither #1890, nor #1902 solve this

PHP_INT_SIZE is 4, I double checked this, just in case ...

@derkostka
Copy link
Contributor Author

Just for your information. The same issue occurs on the last stable version of owncloud as well as the daily build (which includes the mentioned patches).

@derkostka
Copy link
Contributor Author

derkostka commented Apr 5, 2017

Perfect @rikmeijer @nickvergessen , this fixed the issue. I will 👍 on the change, as i do not see any side effect. Please vote for this change in order to get the 32 Bit download right.

derkostka added a commit to derkostka/sabre-http that referenced this issue Apr 9, 2017
Replace stream_copy_to_stream with a chunked process, to handle file sizes hat do not fit into an integer.
Implementation according to http://php.net/manual/en/function.stream-copy-to-stream.php#98119, proposed and implemented by https://github.com/rikmeijer in nextcloud/server#1707 (comment).
@staabm
Copy link
Contributor

staabm commented Apr 9, 2017

Filed a php bug https://bugs.php.net/bug.php?id=74395

@nickvergessen
Copy link
Member

Oh nice work @rikmeijer

@staabm
Copy link
Contributor

staabm commented Apr 10, 2017

Why do you use 32bit php in the first place?

@derkostka
Copy link
Contributor Author

Most SBCs such as Raspberry Pi and Odroid devices still rely on 32 Bit. Therefore this change is a real big thing for those.

@hrlmartins
Copy link

YEP! Just stumbled in this right now. Got really sad :(

@staabm
Copy link
Contributor

staabm commented Jun 12, 2017

just released https://github.com/fruux/sabre-http/releases/tag/4.2.3 which contains a workaround for the 32bit file size problems

@derkostka
Copy link
Contributor Author

derkostka commented Jun 22, 2017

I just checked on daily, big downloads on 32 Bit work perfectly now (6,8 GB file in my testcase), the md5 sum matches and network speed did not slow down (tested in my local lan).

Thank you - Let consider this fixed with Nextcloud 13 💯 👍

@sturze
Copy link

sturze commented Sep 25, 2017

i use Nextcloud 12.0.3 on a 32Bit 16.04 Ubuntu (from a Webhoster) with PHP 7.0.22
i am facing the issue that the zip generated by the Download-Button crashed downloading arround 4GB.

i also tried the sabre-http fix (copied the lib to nextcloud/3rdparty/sabre/http/lib) without success 😢
what did i do wrong?!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.