Enable large file support (on 32bit systems)

[tristanlins]

I run into some problems, with nextcloud on my odroid xu4 (arm32v7) and storing large files (more then 7gb) in the cloud. PHP failed with "failed to open stream: Value too large for defined data type" errors. There are some people out there, having similar problems (like nextcloud/server#1707 or nextcloud/docker#613).

When PHP is compiled with CFLAGS="-D_FILE_OFFSET_BITS=64" (Documentation) it is able to open large files on 32bit systems.

I tested it successfully with the stretch variant (build the official nextcloud image against a modified version of the php 7.3 stretch image). My nextcloud is now able to sync and deliver files larger than 4gb.

There are still a lot of 32bit systems out there.
Especially SoC board like raspberry pi (up to model 2) or odroid xu4.

The patch is fully forward compatible, since the FILE_OFFSET_BITS does not have any effects on 64bit systems:

On 64 bit systems this macro has no effect since the *64 functions are identical to the normal functions.

[tianon]

I tried to find some reference to actual "large file" support in PHP, and the best I found was https://bugs.php.net/bug.php?id=45942, wherein https://bugs.php.net/bug.php?id=45942#1220036600 especially makes me worried about enabling this:

BTW.. it is known that large file support may not work currenlty with PHP.

I'm not sure where to look for further details on this (especially curious is the "it is known" bit -- known where and to whom? the upstream developers in general?) 😕 😞

[tristanlins]

The last comment says:

To this day, even with PHP 5.3.6, the only way to get true large file support
seems to be adding the -D_FILE_OFFSET_BITS=64 compile flag, which still causes a
50% runtime performance penalty.

I did not notice any performance losses.
But I'm not a reference with my test.

As far as I understand _FILE_OFFSET_BITS, it is more like a "hack" with some edge cases.
It worked for me, but if it causes more problems than solves, I'll have to think of something else 🤔

[delfer]

LGTM

[tianon]

This is not a general solution to the problem, and it is actively discouraged by PHP maintainers. As such, I'm closing this PR. Thanks for the contribution and the constructive dialogue/research, @tristanlins! If you find a better solution the PHP developers aren't opposed to, please let us know. 👍

[tristanlins]

I'm okay with this. It is a limitation of 32bit architecture.
I already moved my nextcloud to an Intel Atom based platform :-)

[delfer]

@tianon It was last one chance for all Raspberry Pi and other boards. My nextcloud on Odroid HC2 still can not share files over 2Gb 😢

[tristanlins]

Since the Raspberry 2 Mod. B v1.2 an ARMv8 64bit CPU is installed.
All current Raspberry models should be able to execute a 64bit PHP.
But it is true that unfortunately a lot of SoC boards with 32bit CPU are on the market.

[delfer]

Yes and no. Modern Raspberry Pi boards have 64-bit CPU, but still 32-bit OS.
From my Pi 3:

$ file /usr/bin/sudo
/usr/bin/sudo: setuid ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=02ae0f4b730edb6e01fb34a818cd7ee4a58fc033, stripped

And now here is no stable 64-bit OS for Raspberry in case of proprietary CPU.

[tristanlins]

As I said, it should be able.
But somehow I do not understand why there is no corresponding OS for a 64bit platform. :-(

[delfer]

@tianon from official PHP documentation:

if you want PHP to support LFS (large files) on Linux, then you need to have a recent glibc and you need compile PHP with the following compiler flags: -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64

https://www.php.net/manual/en/intro.filesystem.php

Please, review. We need it!

[tianon]

That's a positive sign for sure, although this comment still worries me: https://bugs.php.net/bug.php?id=45942#1323467326

... adding the -D_FILE_OFFSET_BITS=64 compile flag ... still causes a 50% runtime performance penalty.

[tristanlins]

I can not confirm that setting the option leads to a fundamental slowdown. But for certain commands, e.g. file access is likely to lead to a slowdown. Whether it is really consistently 50%, but I think that is questionable.

[delfer]

@tianon @tristanlins it does not affect performance.
I've made tests and open new PR #901