Remove type for size arg on Streamer constructor #16636

artonge · 2019-08-02T11:15:52Z

On 32 bit systems integers can not be bigger than 2 147 483 647. This is problematic as it throws an error when downloading a folder bigger than 2Go because php will cast the size to float.

scriptator · 2019-09-24T07:53:38Z

Can somebody review and merge this? For me it seems fine and fixes the problem which I experienced

nickvergessen

Urgh, while I can see this solving the problem you have, I guess the issue will pop in quite a lot other locations too. I just think that at some point we just can not manage to stay compatible with 16-bit anymore.

kesselb · 2019-09-24T13:20:47Z

the issue will pop in quite a lot other locations too.

Yes. Another report for a similar problem is: #13160. Unfortunately https://wiki.php.net/rfc/bigint is still work in progress.

A) Add a library like https://github.com/brick/math and use BigInteger for size attributes.
B) Don't type hint $size

I don't like A nor B 🙈 But there are many raspi instances out there.

nickvergessen · 2019-09-24T13:45:05Z

Well, you forgot:
C) Don't support mutli GB files on those rapsi instances 🙈

scriptator · 2019-09-25T16:34:18Z

Urgh, while I can see this solving the problem you have, I guess the issue will pop in quite a lot other locations too.

Totally agree! It would be certainly better to fix the problem at the root

I just think that at some point we just can not manage to stay compatible with 16-bit anymore.

What does this have to do with 16 bit? IMO raspi is running 32 bit which also makes sense when you consider the 2gb limit

kesselb · 2019-09-25T20:08:13Z

Totally agree! It would be certainly better to fix the problem at the root

Where is the root? PHP not having BigInt? Raspian not using 64 bit? We don't have much options. Change the type of $size for every occurence to this BigInteger library will take months. Not adding type hints for $size is the only way I can think of.

What does this have to do with 16 bit? IMO raspi is running 32 bit which also makes sense when you consider the 2gb limit

Typo ...

scriptator · 2019-09-26T13:53:59Z

There is of course also another option: officially drop support for 32 bit systems. As I just found out there are 64 bit OS available for the raspberry and even the Raspberrypi foundation is now working on a 64 bit raspbian kernel. If it is clearly stated everywhere that 32 bit does not work properly and raspbian users should install a 64 bit distro I think it should also be okay

artonge · 2019-09-27T17:16:45Z

I think dropping support for 32-bit systems might be a bit precipitated. 64-bit CPU on a Raspberrypi only came with the version 4, so there will still be a lot of 32-bit rpi for a some years. And the rpi is not the only 32-bit single board computer that is used as a small home server.

So dropping 32-bit support would mean dropping support for a lot of self-hosters.

kesselb · 2019-10-09T14:21:35Z

cc @rullzer 🏓

nachoparker · 2019-12-19T17:35:51Z

Raspbian images are still 32 bits userland, even those with a 64 bits kernel.

kesselb · 2019-12-19T17:37:31Z

Sorry! I forgot the milestone 🙈

kesselb · 2019-12-19T17:39:34Z

lib/private/Streamer.php

@@ -43,7 +43,7 @@ class Streamer {
 	 * @param int $numberOfFiles The number of files (and directories) that will
 	 *        be included in the streamed file
 	 */
-	public function __construct(IRequest $request, int $size, int $numberOfFiles){
+	public function __construct(IRequest $request, $size, int $numberOfFiles){


We should probably add a short note to the docblock why typehinting $size is bad.

go2sh · 2019-12-19T19:26:59Z

I dont think you can change the type to float at all. I had a similar discussions with Morris and the problem is, that you cannot represent every integer with a float exactly especially for big numbers. You might get a value that is some where around the actual size creating a completely new bunch of problems. Out of bound reads, corrupted files, changing files sizes, plus the behavior might be total random as it depends on the actual file size. Because of that, a size in nextcloud should never be float with the consequences for 32 bit instances.

kesselb · 2019-12-19T19:38:18Z

size creating a completely new bunch of problems. Out of bound reads, corrupted files, changing files sizes, plus the behavior might be total random as it depends on the actual file size.

Probably. But this pr is not about making $size a float. It's about removing the type hint. Requested the storage experts for review ;)

rullzer · 2020-01-07T20:03:00Z

Yep, we need to think of a better way to tackle this.
As removing it will also blow up in several cases as the resulting zip file will be invalid.

artonge · 2020-02-24T09:52:55Z

Hi,

Maybe the real solution would be to display an error message to the user explaining why the file or zipped folder can't be downloaded.

Could an exception be thrown here, and caught in an error handler to display the correct error message ?

skjnldsv · 2020-04-10T08:11:44Z

Ping! :)

skjnldsv · 2020-04-10T10:21:09Z

Yep, we need to think of a better way to tackle this.
As removing it will also blow up in several cases as the resulting zip file will be invalid.

What shall we do then?

Wolfgang1966 · 2020-10-24T10:39:22Z

How about using tar archives für everything above 4GB? It is already there depending on the client os and it will at least allow to download big archives at all. Ok, Windows does not support tar archives directly, but there are enough tools out there which do. And it is better than being not able to download the file at all.

Tried it with a 9GB archive, was immediately asked to search for a application in the MS app store which offers several of them.

Consider this a workaround, but at least it is (in my opinion) a fair improvement compared to the current situation.

Wolfgang1966 · 2020-12-07T13:17:50Z

How about using tar archives für everything above 4GB? It is already there depending on the client os and it will at least allow to download big archives at all. Ok, Windows does not support tar archives directly, but there are enough tools out there which do. And it is better than being not able to download the file at all.

Tried it with a 9GB archive, was immediately asked to search for a application in the MS app store which offers several of them.

Consider this a workaround, but at least it is (in my opinion) a fair improvement compared to the current situation.

My version of the constructor in Streamer.php now is:

                if ($size > 0 && $size < 4 * 1000 * 1000 * 1000 && $numberOfFiles < 65536) {
                        $this->streamerInstance = new ZipStreamer(['zip64' => false]);
                } else {
                        $this->streamerInstance = new TarStreamer();
                }

go2sh · 2020-12-08T10:36:20Z

@Wolfgang1966 The problem here is, that the size as float doesn't represent the actual size of the files, as floating point numbers (32bit) are only an aproximation of integers (32bit).

Wolfgang1966 · 2020-12-08T11:57:45Z

@Wolfgang1966 The problem here is, that the size as float doesn't represent the actual size of the files, as floating point numbers (32bit) are only an aproximation of integers (32bit).

This is true, but in that case not relevant. As you can see, the parameter is only used to distuinguish between the usage of 32bit ZIP or anything else. 32bit zip can be used until 4GiB, which is roughly 4.29 GB. As we compare the value to 4.000.000.000 4GB, there is more than enough safety margin for any kind of missing accurancy that might be introduced by using float here.

kesselb · 2020-12-08T15:46:23Z

public function __construct(IRequest $request, int $size, int $numberOfFiles) {

This code fails if you are on a 32bit operating system and size is bigger than 2147483648 because a bigger value is casted to float and then does not match the type hint anymore and 💥

Whatever you are talking about is a different story and unrelated to this pull request.

MichaIng · 2021-09-25T12:33:11Z

Btw, Raspberry Pi 2 v1.2 is already ARMv8 capable: https://en.wikipedia.org/wiki/Raspberry_Pi#Specifications
So relevant 32-bit only SBCs are especially the RPi Zero models. ~~But the Raspberry Pi OS 64-bit version is still in beta, so~~ dropping 32-bit support would still affect most RPi users and of course it would also affect a lot of other SBCs, though with a significantly lower user number.

Not representative, but the top 10 systems running DietPi and participating our survey program:

RPi 4 Model B (aarch64)	3607	20%
RPi 4 Model B (armv7l)	1774	10%
RPi 3 Model B (armv7l)	1252	7%
Native PC (x86_64)	1183	6%
RPi Zero W (armv6l)	1179	6%
RPi 3 Model B (aarch64)	1113	6%
Virtual Machine (x86_64)	894	5%
RPi 3 Model B+ (armv7l)	846	4%
RPi 3 Model B+ (aarch64)	778	4%
RPi 2 Model B (armv7l)	711	4%
RPi B (armv6l)	659	3%

armv7l means 32-bit OS version, ~~which is still the majority due to limitations of the 64-bit RPi userland and GPU accelerated software builds.~~

Resolves #12422 and #15117 On 32 bit systems integers can not be bigger than 2 147 483 647. This is problematic as it throws an error when downloading a folder bigger than 2Go because php will cast the size to float. Signed-off-by: Louis Chemineau <louis@chmn.me>

MichaIng · 2022-02-26T15:19:12Z

Linking the very same issue with the trash bin: #13160
Possible solution is the same.

szaimen · 2022-07-13T09:02:14Z

Closing due to nextcloud/documentation#9071

gary-kim added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Aug 10, 2019

kesselb added 3. to review Waiting for reviews and removed 0. Needs triage Pending check for reproducibility or if it fits our roadmap labels Sep 24, 2019

kesselb requested review from danxuliu, rullzer and nickvergessen September 24, 2019 10:46

nickvergessen approved these changes Sep 24, 2019

View reviewed changes

kesselb approved these changes Sep 25, 2019

View reviewed changes

kesselb mentioned this pull request Sep 28, 2019

fixed bug: type error on large zip download, argument 2 passed to OC\… #15210

Closed

kesselb mentioned this pull request Oct 9, 2019

Trash bin: AbstractTrash::getSize() must be of the type integer, float returned #13160

Closed

kesselb mentioned this pull request Oct 20, 2019

Remove type hint for AbstractTrash.getSize() #17609

Closed

kesselb mentioned this pull request Oct 31, 2019

NC16.0.3 Upload fails with files 2GiB in 32bit instances (again) #16431

Closed

kesselb added this to the Nextcloud 18 milestone Dec 19, 2019

kesselb reviewed Dec 19, 2019

View reviewed changes

rullzer mentioned this pull request Dec 19, 2019

18 beta 4 #18480

Merged

18 tasks

kesselb requested review from icewind1991 and blizzz December 19, 2019 19:38

rullzer mentioned this pull request Jan 2, 2020

18 RC 2 #18639

Merged

1 task

rullzer modified the milestones: Nextcloud 18, Nextcloud 19 Jan 7, 2020

This was referenced Apr 4, 2020

19 beta 2 #20303

Merged

19 beta 3 #20386

Merged

skjnldsv added 2. developing Work in progress and removed 3. to review Waiting for reviews labels Apr 11, 2020

This was referenced Apr 15, 2020

19 beta 4 #20506

Merged

19 Beta 5 #20510

Merged

rullzer mentioned this pull request Apr 23, 2020

19 beta 6 #20611

Merged

11 tasks

rullzer removed this from the Nextcloud 19 milestone Apr 24, 2020

Vigasto mentioned this pull request Sep 21, 2020

Error assembling chunks from large uploads (>4GB) #22936

Closed

kesselb mentioned this pull request Oct 7, 2020

Fix file size computation on 32bit platforms #23257

Merged

santa-klaus mentioned this pull request Oct 19, 2020

Can't download large folders (~>1GB) - Error "Argument 2 passed to OC\Streamer::__construct() must be of the type integer, float given..." #12422

Closed

MichaIng linked an issue Feb 26, 2022 that may be closed by this pull request

TypeError: Argument 2 passed to OC\Streamer::__construct() must be of the type integer, float given, called in /config/www/nextcloud/lib/private/legacy/files.php on line 166 #15117

Closed

fwsmit mentioned this pull request Jul 11, 2022

Fix big file download for 32-bit servers #33197

Closed

szaimen closed this Jul 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove type for size arg on Streamer constructor #16636

Remove type for size arg on Streamer constructor #16636

artonge commented Aug 2, 2019

scriptator commented Sep 24, 2019

nickvergessen left a comment

kesselb commented Sep 24, 2019

nickvergessen commented Sep 24, 2019

scriptator commented Sep 25, 2019

kesselb commented Sep 25, 2019

scriptator commented Sep 26, 2019

artonge commented Sep 27, 2019

kesselb commented Oct 9, 2019

nachoparker commented Dec 19, 2019

kesselb commented Dec 19, 2019

kesselb Dec 19, 2019

go2sh commented Dec 19, 2019

kesselb commented Dec 19, 2019 •

edited

Loading

rullzer commented Jan 7, 2020

artonge commented Feb 24, 2020

skjnldsv commented Apr 10, 2020

skjnldsv commented Apr 10, 2020

Wolfgang1966 commented Oct 24, 2020

Wolfgang1966 commented Dec 7, 2020

go2sh commented Dec 8, 2020

Wolfgang1966 commented Dec 8, 2020

kesselb commented Dec 8, 2020

MichaIng commented Sep 25, 2021 •

edited

Loading

MichaIng commented Feb 26, 2022 •

edited

Loading

szaimen commented Jul 13, 2022

Remove type for size arg on Streamer constructor #16636

Remove type for size arg on Streamer constructor #16636

Conversation

artonge commented Aug 2, 2019

scriptator commented Sep 24, 2019

nickvergessen left a comment

Choose a reason for hiding this comment

kesselb commented Sep 24, 2019

nickvergessen commented Sep 24, 2019

scriptator commented Sep 25, 2019

kesselb commented Sep 25, 2019

scriptator commented Sep 26, 2019

artonge commented Sep 27, 2019

kesselb commented Oct 9, 2019

nachoparker commented Dec 19, 2019

kesselb commented Dec 19, 2019

kesselb Dec 19, 2019

Choose a reason for hiding this comment

go2sh commented Dec 19, 2019

kesselb commented Dec 19, 2019 • edited Loading

rullzer commented Jan 7, 2020

artonge commented Feb 24, 2020

skjnldsv commented Apr 10, 2020

skjnldsv commented Apr 10, 2020

Wolfgang1966 commented Oct 24, 2020

Wolfgang1966 commented Dec 7, 2020

go2sh commented Dec 8, 2020

Wolfgang1966 commented Dec 8, 2020

kesselb commented Dec 8, 2020

MichaIng commented Sep 25, 2021 • edited Loading

MichaIng commented Feb 26, 2022 • edited Loading

szaimen commented Jul 13, 2022

kesselb commented Dec 19, 2019 •

edited

Loading

MichaIng commented Sep 25, 2021 •

edited

Loading

MichaIng commented Feb 26, 2022 •

edited

Loading