-
-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can't download large folders (~>1GB) - Error "Argument 2 passed to OC\Streamer::__construct() must be of the type integer, float given..." #12422
Comments
GitMate.io thinks possibly related issues are #8964 (Argument 2 passed to OC_App::isAppCompatible() must be of the type array, null given), #10601 (TypeError: Argument 8 passed to OCA\Files_Sharing\Controller\ShareAPIController::__construct() must be of th...), #3376 (Argument 3 passed to OCA\\Files\\Service\\TagService::__construct() must be an instance of OCP\\ITags), #2297 (Argument 1 passed to OC\Files\Config\CachedMountInfo::__construct() must implement interface OCP\IUser, null given), and #8489 (cannot download large folder with lots of small files). |
PHP_INT_MAX is 2147483647 for 32-bit. Looks like a value like 12288299774 is casted to float when greater than int max. I guess we could allow integer or float as value. Just wondering why you run a 32-bit linux on hardware with more than 4gb memory? |
server/lib/private/Streamer.php Line 46 in 2e36069
Does it work when you change |
Hi @danielkesselberg ! Thanks for your help. I have tried that, but for a folder with 4,2 GB size I get an incomplete .tar file. I can extract almost all files from it but then I get an error. I'm using 32-bit with 2 GB of RAM, not 4 GB. |
I think you meet the hard limit of the 32bit system (maybe not easy to resolve). If you are dealing with large files, it's better to have a 64bit system. |
Same here. It seems like I can download folders up to 2 GB, which is my RAM size, and cannot download anything larger than this. Mint 18.3 x32 |
Slackware 14.2 x86_64 |
Operating System: Raspbian GNU/Linux 9 (stretch)
EDIT : I agree with wxiaoguang. That's a problem linked to 32bits systems. And since Raspbian is not available as 64bits we run a 32-bits apache/php instance... |
Same with NC 16. Operating System: Ubuntu 16.04, arm7l
|
I just updated from owncloud to the newest version of nextcloud (16.0.4) and experience the same issue. Is there anything I can do? Raspbian GNU/Linux 9
|
This commit fixes the problem for me: https://github.com/artonge/server/commit/435022515de1983f0fe3d3116acb71a0ed439693 |
Hi, |
Same on my debian 32bits system the patch fixed the problem |
Large directories not downloading in 18.0.6: artonge@4350225 scheint auch bei 18.0.6 zu helfen. |
I have the same problem in 19.0.2 running on 32-bit RaspberryOS 10.4. It can't download a folder containing 380 photos (~3.2 GB). If I apply https://github.com/artonge/server/commit/435022515de1983f0fe3d3116acb71a0ed439693 it works smoothly. It would be great if this fix could be included in the next update. |
I confirm issue in 19.0.3 running on 32-bit Raspbian 10 (buster) Look very similar to #15117 |
I also have this problem on a 32-bit Armbian on an OdroidXU4. |
Same for me, Odroid HC1 (=XU4) with current Ubuntu. Using PHP 7.4 Fast CGI Module. |
The fix seems to be a one-liner basically: In lib/private/Streamer.php exchange public function __construct(IRequest $request, float $size, int $numberOfFiles) (line 56 in Nextcloud 19.0.3). Didn't test that in depth, but at least got rid of the error message. The reason basically seems to be that PHP implicitely converts int to float as soon as a number overflow is about to occur. See https://www.php.net/manual/de/language.types.integer.php |
Have it now running this way fopr more than two weeks, no problems. Would recommend to use that fix. Do I really have to open a PR for a one-liner change? |
@Wolfgang1966 This is awesome news! I hope it gets fixed ASAP. |
@Wolfgang1966, not sure if floats can be safely used for size, their precision is limited. But I'm not a php dev. Anyway, I did a bounty for this some time ago: https://www.bountysource.com/issues/66077825-can-t-download-large-folders-1gb-error-argument-2-passed-to-oc-streamer-__construct-must-be-of-the-type-integer-float-given, so whoever gets their PR merged... |
@Wolfgang1966 @nfp0 |
@santa-klaus Oh, I'm sorry. I missed that. |
As far as I understand the code the number passed as argument is only used to decide what kind of archive to use, zip32 or zip64. So there is no real need to have a exact value. In principle one could just leave away the logic totally and always use zip64 as default, as far as there is no drawback known that may exist on client side. |
@Wolfgang1966 Can Zip64 be used on 32bit machines? |
Did a few more tests and found out that indeed Zip files above 4GB will be corrupted. Didn't notice that earlier because my previous tests were with folders above 2GB but below 4GB. Changed the code now the following way:
Above 4GB the archive format is tar instead of zip that way. While there is no on board support for tar archives in Windows, there are may tools around providing support, and many users might have one installed already. If not, Windows 10 immediately offers to install one from the Microsoft store (at least it did that on my first attempt with a 9GB tar archive), so it should not be too difficult for the users to handle the archive. From my point of view this is an improvement compared to not being able to download it at all. |
By "download", do you mean download the full folder from the Web UI ? Or does this problem also occurs when a user synchronizes a folder that was shared to her ? Then that would be a big blocker for the use of NC in offices ... |
AFAIK this problem only affects downloads. There on the fly an archive (ZIP or TAR) is created, which then leads to two problems:
Yours sincerely Wolfgang |
Thanks for confirming. I could then resolve the problem of big folders not downloading simply by :
There are so many server-side settings showing in Google when searching solutions that we overlooked the "normal" default client behavior. All good now. |
When a folder larger than ~2GB is downloaded, zip32 doesn't cut it. Since zip64 doesn't work on 32-bit clients, fall back to generating a tarball. Fixes nextcloud#12422 and nextcloud#15117
When a folder larger than ~2GB is downloaded, zip32 doesn't cut it. Since zip64 doesn't work on 32-bit clients, fall back to generating a tarball. Fixes nextcloud#12422 and nextcloud#15117 Signed-off-by: fwsmit <fw.smit01@gmail.com>
Closing due to nextcloud/documentation#9071 |
Steps to reproduce
Expected behaviour
Zip file should be generated correctly and downloaded correctly.
Actual behaviour
An error is thrown to the user with a reqId to request support.
Server configuration detail
Operating system: Linux 4.9.0-8-686-pae #1 SMP Debian 4.9.130-2 (2018-10-27) i686
Webserver: nginx/1.10.3 (fpm-fcgi)
Database: mysql 10.1.26
PHP version:
7.0.30-0+deb9u1
Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, session, standard, cgi-fcgi, mysqlnd, PDO, xml, apcu, apc, calendar, ctype, curl, dom, mbstring, fileinfo, ftp, gd, gettext, iconv, igbinary, imagick, imap, intl, json, exif, mcrypt, mysqli, pdo_mysql, pdo_sqlite, Phar, posix, readline, redis, shmop, SimpleXML, sockets, sqlite3, sysvmsg, sysvsem, sysvshm, tokenizer, wddx, xmlreader, xmlwriter, xsl, zip, Zend OPcache
Nextcloud version: 14.0.3 - 14.0.3.0
Updated from an older Nextcloud/ownCloud or fresh install:
Where did you install Nextcloud from: Nextcloud official web
Signing status
List of activated apps
Configuration (config/config.php)
Are you using external storage, if yes which one: No
Are you using encryption: No
Are you using an external user-backend, if yes which one: No
Client configuration
Browser: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36
Operating system: Mac OS X Mojave 10.14.1
Logs
Browser log
Nextcloud log
Browser log
I have tried with several folders, but I always get the same problem. Notice that is a 32-bit server with latest Debian version.
The text was updated successfully, but these errors were encountered: