Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't download large folders (~>1GB) - Error "Argument 2 passed to OC\Streamer::__construct() must be of the type integer, float given..." #12422

Closed
alejandrocq opened this issue Nov 12, 2018 · 32 comments
Labels
3. to review Waiting for reviews bug

Comments

@alejandrocq
Copy link

alejandrocq commented Nov 12, 2018

Steps to reproduce

  1. Try to download a large folder (>1GB).
  2. Error is thrown while trying to generate a zip file.

Expected behaviour

Zip file should be generated correctly and downloaded correctly.

Actual behaviour

An error is thrown to the user with a reqId to request support.

Server configuration detail

Operating system: Linux 4.9.0-8-686-pae #1 SMP Debian 4.9.130-2 (2018-10-27) i686

Webserver: nginx/1.10.3 (fpm-fcgi)

Database: mysql 10.1.26

PHP version:

7.0.30-0+deb9u1
Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, session, standard, cgi-fcgi, mysqlnd, PDO, xml, apcu, apc, calendar, ctype, curl, dom, mbstring, fileinfo, ftp, gd, gettext, iconv, igbinary, imagick, imap, intl, json, exif, mcrypt, mysqli, pdo_mysql, pdo_sqlite, Phar, posix, readline, redis, shmop, SimpleXML, sockets, sqlite3, sysvmsg, sysvsem, sysvshm, tokenizer, wddx, xmlreader, xmlwriter, xsl, zip, Zend OPcache

Nextcloud version: 14.0.3 - 14.0.3.0

Updated from an older Nextcloud/ownCloud or fresh install:

Where did you install Nextcloud from: Nextcloud official web

Signing status
List of activated apps
Enabled:
 - accessibility: 1.0.1
 - activity: 2.7.0
 - calendar: 1.6.3
 - cloud_federation_api: 0.0.1
 - comments: 1.4.0
 - contacts: 2.1.6
 - dav: 1.6.0
 - federatedfilesharing: 1.4.0
 - federation: 1.4.0
 - files: 1.9.0
 - files_external: 1.5.0
 - files_pdfviewer: 1.3.2
 - files_sharing: 1.6.2
 - files_texteditor: 2.6.0
 - files_trashbin: 1.4.1
 - files_versions: 1.7.1
 - files_videoplayer: 1.3.0
 - firstrunwizard: 2.3.0
 - gallery: 18.1.0
 - issuetemplate: 0.4.0
 - logreader: 2.0.0
 - lookup_server_connector: 1.2.0
 - nextcloud_announcements: 1.3.0
 - notes: 2.4.2
 - notifications: 2.2.1
 - oauth2: 1.2.1
 - password_policy: 1.4.0
 - provisioning_api: 1.4.0
 - serverinfo: 1.4.0
 - sharebymail: 1.4.0
 - support: 1.0.0
 - survey_client: 1.2.0
 - systemtags: 1.4.0
 - tasks: 0.9.7
 - theming: 1.5.0
 - twofactor_backupcodes: 1.3.1
 - updatenotification: 1.4.1
 - workflowengine: 1.4.0
Disabled:
 - admin_audit
 - encryption
 - user_external
 - user_ldap

Configuration (config/config.php)
{
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        "***REMOVED SENSITIVE VALUE***"
    ],
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
    "dbtype": "mysql",
    "version": "14.0.3.0",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbtableprefix": "oc_",
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "logtimezone": "UTC",
    "installed": true,
    "loglevel": 0,
    "memcache.local": "\\OC\\Memcache\\APCu",
    "maintenance": false,
    "theme": "",
    "mail_smtpmode": "smtp",
    "mail_smtpauthtype": "PLAIN",
    "mail_smtpsecure": "tls",
    "mail_from_address": "***REMOVED SENSITIVE VALUE***",
    "mail_domain": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpauth": 1,
    "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpport": "587",
    "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
    "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
    "updater.secret": "***REMOVED SENSITIVE VALUE***"
}

Are you using external storage, if yes which one: No

Are you using encryption: No

Are you using an external user-backend, if yes which one: No

Client configuration

Browser: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36

Operating system: Mac OS X Mojave 10.14.1

Logs

Browser log

Nextcloud log
TypeError: Argument 2 passed to OC\Streamer::__construct() must be of the type integer, float given, called in /var/www/html/owncloud/lib/private/legacy/files.php on line 166
/var/www/html/owncloud/lib/private/legacy/files.php - line 166:

OC\Streamer->__construct(OC\AppFramework\Http\Request {}, 12288299774, 893)

/var/www/html/owncloud/apps/files/ajax/download.php - line 64:

OC_Files::get("__SENSITIVE_INFO_HIDDEN__", "__SENSITIVE_INFO_HIDDEN", { head: false})

/var/www/html/owncloud/lib/private/Route/Route.php - line 155:

require_once("/var/www/ht ... p")


OC\Route\Route->OC\Route\{closure}("*** sensiti ... *")

/var/www/html/owncloud/lib/private/Route/Router.php - line 297:

call_user_func(Closure {}, { _route: "f ... "})

/var/www/html/owncloud/lib/base.php - line 987:

OC\Route\Router->match("/apps/files/ajax/download.php")

/var/www/html/owncloud/index.php - line 42:

OC::handleRequest()
Browser log

I have tried with several folders, but I always get the same problem. Notice that is a 32-bit server with latest Debian version.

@nextcloud-bot
Copy link
Member

GitMate.io thinks possibly related issues are #8964 (Argument 2 passed to OC_App::isAppCompatible() must be of the type array, null given), #10601 (TypeError: Argument 8 passed to OCA\Files_Sharing\Controller\ShareAPIController::__construct() must be of th...), #3376 (Argument 3 passed to OCA\\Files\\Service\\TagService::__construct() must be an instance of OCP\\ITags), #2297 (Argument 1 passed to OC\Files\Config\CachedMountInfo::__construct() must implement interface OCP\IUser, null given), and #8489 (cannot download large folder with lots of small files).

@kesselb
Copy link
Contributor

kesselb commented Nov 12, 2018

PHP_INT_MAX is 2147483647 for 32-bit. Looks like a value like 12288299774 is casted to float when greater than int max. I guess we could allow integer or float as value. Just wondering why you run a 32-bit linux on hardware with more than 4gb memory?

@kesselb
Copy link
Contributor

kesselb commented Nov 12, 2018

public function __construct(IRequest $request, int $size, int $numberOfFiles){

Does it work when you change int $size to $size?

@alejandrocq
Copy link
Author

Hi @danielkesselberg ! Thanks for your help. I have tried that, but for a folder with 4,2 GB size I get an incomplete .tar file. I can extract almost all files from it but then I get an error.

I'm using 32-bit with 2 GB of RAM, not 4 GB.

@wxiaoguang
Copy link

I think you meet the hard limit of the 32bit system (maybe not easy to resolve). If you are dealing with large files, it's better to have a 64bit system.

@sertraline
Copy link

Same here. It seems like I can download folders up to 2 GB, which is my RAM size, and cannot download anything larger than this. Mint 18.3 x32

@myevit
Copy link

myevit commented Jan 7, 2019

screen shot 2019-01-07 at 02 22 55

> Same here. It seems like I can download folders up to 2 GB, which is my RAM size, and cannot download anything larger than this. Mint 18.3 x32

same here - Physical Memory Usage 1.91 GB / 2 GB (95.59%) during downloading 5GB file then random drop on 500mb - 1.6gb

@StreamThreader
Copy link

Slackware 14.2 x86_64
Nextcloud 15.0.5
If download from local net (1Gbit link), all ok
If download through internet 50Mbit link, zip drop on 1GB (folder larger than 1.5GB).
HDD and RAM have free space.
Just files larger than 2GB downloaded without problem.
This problem with ZIP stream

@Elerir
Copy link

Elerir commented Apr 26, 2019

Operating System: Raspbian GNU/Linux 9 (stretch)
Kernel: Linux 4.14.98-v7+
Nextcloud 15.0.7
Doesn't seem linked to RAM.. my raspberry still has >60% free RAM when nextcloud displays the error and cancels the request

{"Exception":"TypeError","Message":"Argument 2 passed to OC\\Streamer::__construct() must be of the type integer, float given, called in \/nas\/data\/nextcloud\/lib\/private\/legacy\/files.php on line 166","Code":0,"Trace":[{"file":"\/nas\/data\/nextcloud\/lib\/private\/legacy\/files.php","line":166,"function":"__construct","class":"OC\\Streamer","type":"->","args":[{"__class__":"OC\\AppFramework\\Http\\Request"},13732257759,7612]},{"file":"\/nas\/data\/nextcloud\/apps\/files\/ajax\/download.php","line":64,"function":"get","class":"OC_Files","type":"::","args":["\/","Toutes photos",{"head":false}]},{"file":"\/nas\/data\/nextcloud\/lib\/private\/Route\/Route.php","line":155,"args":["\/nas\/data\/nextcloud\/apps\/files\/ajax\/download.php"],"function":"require_once"},{"function":"OC\\Route\\{closure}","class":"OC\\Route\\Route","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/nas\/data\/nextcloud\/lib\/private\/Route\/Router.php","line":297,"function":"call_user_func","args":[{"__class__":"Closure"},{"_route":"files_ajax_download"}]},{"file":"\/nas\/data\/nextcloud\/lib\/base.php","line":987,"function":"match","class":"OC\\Route\\Router","type":"->","args":["\/apps\/files\/ajax\/download.php"]},{"file":"\/nas\/data\/nextcloud\/index.php","line":42,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"\/nas\/data\/nextcloud\/lib\/private\/Streamer.php","Line":46,"CustomMessage":"--"}

EDIT : I agree with wxiaoguang. That's a problem linked to 32bits systems. And since Raspbian is not available as 64bits we run a 32-bits apache/php instance...

@skjnldsv skjnldsv added the 0. Needs triage Pending check for reproducibility or if it fits our roadmap label Jun 12, 2019
@maranov
Copy link

maranov commented Jul 27, 2019

Same with NC 16.

Operating System: Ubuntu 16.04, arm7l
Kernel: Linux 4.14.111-139
Nextcloud 16.0.3

{"Exception":"TypeError","Message":"Argument 2 passed to OC\\Streamer::__construct() must be of the type integer, float given, called in /var/www/nextcloud/lib/private/legacy/files.php on line 166","Code":0,"Trace":[{"file":"/var/www/nextcloud/lib/private/legacy/files.php","line":166,"function":"__construct","class":"OC\\Streamer","type":"->","args":[{"__class__":"OC\\AppFramework\\Http\\Request"},6974294953,3942]},{"file":"/var/www/nextcloud/apps/files/ajax/download.php","line":64,"function":"get","class":"OC_Files","type":"::","args":["/Photos","SPŠ",{"head":false}]},{"file":"/var/www/nextcloud/lib/private/Route/Route.php","line":155,"args":["/var/www/nextcloud/apps/files/ajax/download.php"],"function":"require_once"},{"function":"OC\\Route\\{closure}","class":"OC\\Route\\Route","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/lib/private/Route/Router.php","line":297,"function":"call_user_func","args":[{"__class__":"Closure"},{"_route":"files_ajax_download"}]},{"file":"/var/www/nextcloud/lib/base.php","line":975,"function":"match","class":"OC\\Route\\Router","type":"->","args":["/apps/files/ajax/download.php"]},{"file":"/var/www/nextcloud/index.php","line":42,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"/var/www/nextcloud/lib/private/Streamer.php","Line":46,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.102 Safari/537.36 Vivaldi/2.6.1566.44","version":"16.0.3.0","id":"5d3c19d192ad0"}

@scriptator
Copy link

scriptator commented Sep 23, 2019

I just updated from owncloud to the newest version of nextcloud (16.0.4) and experience the same issue. Is there anything I can do?

Raspbian GNU/Linux 9
Linux raspberrypi 4.19.66-v7+

TypeError: Argument 2 passed to OC\Streamer::__construct() must be of the type integer, float given, called in /var/www/nextcloud/lib/private/legacy/files.php on line 166

/var/www/nextcloud/lib/private/legacy/files.php - line 166:
OC\Streamer->__construct(OC\AppFramework\Http\Request {}, 3101495978, 9)

/var/www/nextcloud/apps/files/ajax/download.php - line 64:
OC_Files::get("/Fotos", "XXXX", { head: false})

/var/www/nextcloud/lib/private/Route/Route.php - line 155:
require_once("/var/www/ne ... p")

<<closure>>
OC\Route\Route->OC\Route\{closure}("*** sensiti ... *")

/var/www/nextcloud/lib/private/Route/Router.php - line 297:
call_user_func(Closure {}, { _route: "f ... "})

/var/www/nextcloud/lib/base.php - line 975:
OC\Route\Router->match("/apps/files/ajax/download.php")

/var/www/nextcloud/index.php - line 42:
OC::handleRequest()

@scriptator
Copy link

This commit fixes the problem for me: https://github.com/artonge/server/commit/435022515de1983f0fe3d3116acb71a0ed439693

@cquike
Copy link

cquike commented Oct 22, 2019

Hi,
just for the record: I had the same issue also in Raspbian 32 bits with the download link in a shared folder even if it didn't exceed the 2GB limit. Applying the patch mentioned in the last comment also fixed the problem for me.

@skjnldsv skjnldsv added 3. to review Waiting for reviews and removed 0. Needs triage Pending check for reproducibility or if it fits our roadmap labels Apr 10, 2020
@Grokafar
Copy link

Same on my debian 32bits system the patch fixed the problem

@dd5fz
Copy link

dd5fz commented Jun 18, 2020

Large directories not downloading in 18.0.6:

artonge@4350225 scheint auch bei 18.0.6 zu helfen.

@santa-klaus
Copy link

santa-klaus commented Sep 1, 2020

I have the same problem in 19.0.2 running on 32-bit RaspberryOS 10.4. It can't download a folder containing 380 photos (~3.2 GB).

If I apply https://github.com/artonge/server/commit/435022515de1983f0fe3d3116acb71a0ed439693 it works smoothly.

It would be great if this fix could be included in the next update.

@cClaude
Copy link

cClaude commented Sep 24, 2020

I confirm issue in 19.0.3 running on 32-bit Raspbian 10 (buster)

Look very similar to #15117

@nfp0
Copy link

nfp0 commented Sep 24, 2020

I also have this problem on a 32-bit Armbian on an OdroidXU4.

@Wolfgang1966
Copy link

Same for me, Odroid HC1 (=XU4) with current Ubuntu. Using PHP 7.4 Fast CGI Module.

@Wolfgang1966
Copy link

The fix seems to be a one-liner basically:

In lib/private/Streamer.php

exchange
public function __construct(IRequest $request, int $size, int $numberOfFiles)

public function __construct(IRequest $request, float $size, int $numberOfFiles)

(line 56 in Nextcloud 19.0.3).

Didn't test that in depth, but at least got rid of the error message. The reason basically seems to be that PHP implicitely converts int to float as soon as a number overflow is about to occur. See https://www.php.net/manual/de/language.types.integer.php

@Wolfgang1966
Copy link

The fix seems to be a one-liner basically:

In lib/private/Streamer.php

exchange
public function __construct(IRequest $request, int $size, int $numberOfFiles)

public function __construct(IRequest $request, float $size, int $numberOfFiles)

(line 56 in Nextcloud 19.0.3).

Didn't test that in depth, but at least got rid of the error message. The reason basically seems to be that PHP implicitely converts int to float as soon as a number overflow is about to occur. See https://www.php.net/manual/de/language.types.integer.php

Have it now running this way fopr more than two weeks, no problems. Would recommend to use that fix.

Do I really have to open a PR for a one-liner change?

@nfp0
Copy link

nfp0 commented Oct 17, 2020

@Wolfgang1966 This is awesome news! I hope it gets fixed ASAP.
Yes, please open a PR, as that will help the developers. Seems to be a simple fix, but they might not notice your posts.

@maranov
Copy link

maranov commented Oct 17, 2020

@Wolfgang1966, not sure if floats can be safely used for size, their precision is limited. But I'm not a php dev.

Anyway, I did a bounty for this some time ago: https://www.bountysource.com/issues/66077825-can-t-download-large-folders-1gb-error-argument-2-passed-to-oc-streamer-__construct-must-be-of-the-type-integer-float-given, so whoever gets their PR merged...

@santa-klaus
Copy link

santa-klaus commented Oct 19, 2020

@Wolfgang1966 @nfp0
No, you don't have to open a PR, as there is one mentioned earlier in this issue. You can read more details about the current state and the problems with making $size a float there. I am not sure if there is anything planned for this issue at the moment though.
#16636

@nfp0
Copy link

nfp0 commented Oct 19, 2020

@santa-klaus Oh, I'm sorry. I missed that.

@Wolfgang1966
Copy link

As far as I understand the code the number passed as argument is only used to decide what kind of archive to use, zip32 or zip64. So there is no real need to have a exact value. In principle one could just leave away the logic totally and always use zip64 as default, as far as there is no drawback known that may exist on client side.

@nfp0
Copy link

nfp0 commented Oct 24, 2020

@Wolfgang1966 Can Zip64 be used on 32bit machines?

@Wolfgang1966
Copy link

Did a few more tests and found out that indeed Zip files above 4GB will be corrupted. Didn't notice that earlier because my previous tests were with folders above 2GB but below 4GB.

Changed the code now the following way:

                if ($size > 0 && $size < 4 * 1000 * 1000 * 1000 && $numberOfFiles < 65536) {
                        $this->streamerInstance = new ZipStreamer(['zip64' => false]);
//              } elseif ($request->isUserAgent($this->preferTarFor)) {
//                      $this->streamerInstance = new TarStreamer();
                } else {
                        $this->streamerInstance = new TarStreamer();
                }

Above 4GB the archive format is tar instead of zip that way. While there is no on board support for tar archives in Windows, there are may tools around providing support, and many users might have one installed already. If not, Windows 10 immediately offers to install one from the Microsoft store (at least it did that on my first attempt with a 9GB tar archive), so it should not be too difficult for the users to handle the archive. From my point of view this is an improvement compared to not being able to download it at all.

@kitatek
Copy link

kitatek commented Jan 14, 2021

not being able to download it at all

@Wolfgang1966

By "download", do you mean download the full folder from the Web UI ?

Or does this problem also occurs when a user synchronizes a folder that was shared to her ?

Then that would be a big blocker for the use of NC in offices ...

@Wolfgang1966
Copy link

AFAIK this problem only affects downloads. There on the fly an archive (ZIP or TAR) is created, which then leads to two problems:

  1. Problem: The size is passed as INT value, which is 32 bit signed. If the total size of the folder exceeds the max value of a signed 32bit int (2,147,483,647), then PHP implicitely uses a float in the calling function, which leads to the error named in this topic. This can be solved by changing the parameter to float instead. Yes, you loose precision by that, but this precision is not needed here.
  2. Problem: Everything works now as long as the size of the archive is below 4GB (or Nextcloud decides that the client platform prefers TAR over ZIP). When the archive size exceeds 4GB, then the ZIP file created by the streamer will become corrupt, so for archives above 4GB only TAR works correctly.

Yours sincerely

Wolfgang

@kitatek
Copy link

kitatek commented Jan 15, 2021

Thanks for confirming. I could then resolve the problem of big folders not downloading simply by :

  • deactivating the 500MB limit on folder downloads, set in the client GUI settings.

There are so many server-side settings showing in Google when searching solutions that we overlooked the "normal" default client behavior. All good now.

fwsmit added a commit to fwsmit/server that referenced this issue Jul 11, 2022
When a folder larger than ~2GB is downloaded, zip32 doesn't cut it.
Since zip64 doesn't work on 32-bit clients, fall back to generating a
tarball.

Fixes nextcloud#12422 and nextcloud#15117
fwsmit added a commit to fwsmit/server that referenced this issue Jul 11, 2022
When a folder larger than ~2GB is downloaded, zip32 doesn't cut it.
Since zip64 doesn't work on 32-bit clients, fall back to generating a
tarball.

Fixes nextcloud#12422 and nextcloud#15117

Signed-off-by: fwsmit <fw.smit01@gmail.com>
@szaimen
Copy link
Contributor

szaimen commented Jul 13, 2022

Closing due to nextcloud/documentation#9071

@szaimen szaimen closed this as completed Jul 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3. to review Waiting for reviews bug
Projects
None yet