32bit AppImage with size bigger than 2GB not working

[sigmaboy]

When I built an large (5GB) x86 AppImage on my x86_64 openSUSE linux I get the following error when running it:

Cannot open /proc/self/exe: Value too large for defined data type
Failed to get fs offset for /proc/self/exe

It seems my image is too big. A friend of mine is using Ubuntu and for him the generated AppImage works fine.
Since it's a video game with packed size of 5GB I can't get it smaller.
I googled a lot and found out the following. Maybe I need to rebuild some of my software stack (maybe coreutils) with the following gcc option

 -D_FILE_OFFSET_BITS=64

But I think this is already activated on my whole software stack, because this is stuff which changed in like 2006. But of course I'm not sure.

I successfully built x86 appimages (also games) which are smaller than 2GB which work flawlessly.

Strace prints the following line when executing the binary

[...]
open("/proc/self/exe", O_RDONLY)        = -1 EOVERFLOW (Value too large for defined data type)
write(2, "Cannot open /proc/self/exe: Valu"..., 66Cannot open /proc/self/exe: Value too large for defined data type
[...]

I'm using openSUSE 42.3 x86_64 with the default kernel.

If you need any more info, just ask.
Thank you in advance

[probonopd]

Very interesting, thanks for reporting. I was not aware that there is a limit on the size of the executable to access its location from /proc/self/exe.

You are saying that the same large AppImage works on one system (openSUSE) bit not on another (Ubuntu)?

Then possibly you should report this to the openSUSE Bugzilla. I don't think we can fix it from within this project (though I might be wrong).

[TheAssassin]

@probonopd I think we have a similar issue in this tracker or maybe linuxdeployqt's. Might be worth using the search to find it.

[sigmaboy]

You are saying that the same large AppImage works on one system (openSUSE) bit not on another (Ubuntu)?
It's the exact opposite. It doesn't work on openSUSE but it works on Ubuntu.

Thanks for the hint. I'll open an bug on the openSUSE bugzilla. Since it's an commercial game I cannot upload it to public. Maybe I search for a free/FOSS alternative to reproduce the problem.

[TheAssassin]

@sigmaboy you could perhaps try to reproduce the issue with a mock AppImage (if it's a size issue, just put random data in there or so).

[sigmaboy]

I reproduced the error with random data and as I expected i get the same error. Here is the link to the test AppImage with the error:
https://drive.google.com/open?id=1l9NWQqaYhv3kLl8dkUOjixjlPw23zgD2

[probonopd]

Thanks @sigmaboy. Do you have a link to the issue on the openSUSE Bugzilla?

[sigmaboy]

I must correct one thing. (X)Ubuntu 18.04 x86_64 has the same issue

[TheAssassin]

By the way, @sigmaboy:

https://github.com/AppImage/AppImageKit/blob/fc61cbacac6c2dde26fbf531f47c92566c07ae83/src/CMakeLists.txt#L56-L62

[sigmaboy]

I think another component isn't compiled with the right option. But I don't know which so I created the issue here. the bug report doesn't have any progress yet. But the filesize cap is 2GB

[probonopd]

If the error lies with AppImage, why would it then work on some distributions but not others? Could it be that the libraries the AppImage runtime loads from the base system (e.g., libfuse) also would need to be compiled with -D_FILE_OFFSET_BITS=64? (Just blindly guessing here.)

[TheAssassin]

@sigmaboy you might be right, squashfuse etc. may not necessarily be built with this option. But @probonopd's theory sounds reasonable as well. Maybe you can try to mount the AppImage manually? Just sudo mount -o loop my.AppImage /mnt.

[sigmaboy]

the issue gets even weirder. My friend with ubuntu doesn't get the error. My posted AppImage works fine for him. But I installed a (X)ubuntu 17.10 and a 18.04 x86_64 and I get the error from above.

These are the libraries I installed on the Ubuntu vm

ii  gcc-7-base:i386                       7.2.0-8ubuntu3.2                         i386         GCC, the GNU Compiler Collection (base package)
ii  libc6:i386                            2.26-0ubuntu2.1                          i386         GNU C Library: Shared libraries
ii  libgcc1:i386                          1:7.2.0-8ubuntu3.2                       i386         GCC support library
ii  libgpm2:i386                          1.20.4-6.2                               i386         General Purpose Mouse - shared library
ii  libncurses5:i386                      6.0+20160625-1ubuntu1                    i386         shared libraries for terminal handling
ii  libstdc++6:i386                       7.2.0-8ubuntu3.2                         i386         GNU Standard C++ Library v3
ii  libtinfo5:i386                        6.0+20160625-1ubuntu1                    i386         shared low-level terminfo library for terminal handling
ii  zlib1g:i386                           1:1.2.11.dfsg-0ubuntu2                   i386         compression library - runtime

I thought that maybe some coreutils programs causes the issue. But as I wrote in this comment I can reproduce the error on x86_64 ubuntu virtual machine. This is odd because the same AppImage works fine in the Ubuntu x86_64 of my friend.

So maybe this is a missing package error? I only installed base Xubuntu with the i386 libraries from above.

@TheAssassin I can't mount the image as a loop mount. I think I need the offset with ./*.AppImage --appimage-offset but this option cannot be used because it runs into the Cannot open /proc/self/exe: Value too large for defined data type error.

I only can surmise that this a compiling, package or filesystem related issue. I use ext4 as my filesystem. Jfyi

[TheAssassin]

You shouldn't have to specify the offset normally. What error message do you get?

You should try e.g., a live ISO to check whether the issue is limited to your installation.

[sigmaboy]

@TheAssassin

sigmaboy@ubuntu-17:~$ sudo mount -o loop TestApp.AppImage /mnt/
 
mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error.

I'll try a live ISO and test it there. And on my other computer. But I think I will get the same error as before. Since I used a KVM vm with it's own kernel and userland my host system shouldn't have something to do with the error on the ubuntu VM

[TheAssassin]

Ah, you already tested in a VM. Well, then this could really be a bug. Please tell me what exact systems you tried it on (ideally the ISOs' URLs) and I will try to reproduce these issues.

[sigmaboy]

@TheAssassin
Xubuntu 18.04 x86_64
http://cdimage.ubuntu.com/xubuntu/releases/18.04/release/xubuntu-18.04-desktop-amd64.iso
http://cdimage.ubuntu.com/xubuntu/releases/18.04/release/xubuntu-18.04-desktop-amd64.iso.torrent

Xubuntu 17.10 x86_64
http://cdimage.ubuntu.com/xubuntu/releases/17.10.1/release/xubuntu-17.10.1-desktop-amd64.iso
http://cdimage.ubuntu.com/xubuntu/releases/17.10.1/release/xubuntu-17.10.1-desktop-amd64.iso.torrent

openSUSE Leap 15.0 x86_64
DVD Image:
https://download.opensuse.org/distribution/leap/15.0/iso/openSUSE-Leap-15.0-DVD-x86_64.iso
http://download.opensuse.org/distribution/leap/15.0/iso/openSUSE-Leap-15.0-DVD-x86_64.iso.torrent

[TheAssassin]

Looking into this issue at the moment. At first I thought the place where it's missing is the runtime compilation, but actually the flag is there in our build script.

I think OBS embeds the AppRun binary, doesn't it? It's one of the four places where /proc/self/exe is read (aside from binreloc.c, appimagetool.c and runtime.c). Apparently _FILE_OFFSET_BITS isn't defined during the compilation, and I guess this is why your AppImage crashes during startup. I'm going to push a fix for that in a minute, then you can exchange the contained AppRun for an upstream binary and see whether it works then.

[TheAssassin]

It appears like #814 fixed the issue. Please reopen if this is not the case. We don't have any influence when the new AppRun will be available on OBS, please refer to #opensuse-buildservice on Freenode.

[sigmaboy]

@TheAssassin
I still can reproduce this on openSUSE Leap 15.0 x86_64 and Ubuntu 18.04 x86_64 with an virtual machine and the current Continuous build.

Could you please reopen the issue and investigate further. Because you closed it, I cannot re-open it by myself. Only the one who closed it, is able to repon the issue.

And thank you very much for all your help. If I can do anything, just ask

[probonopd]

Thanks for re-testing @sigmaboy

[TheAssassin]

Please provide an AppImage for me to test.

[sigmaboy]

@TheAssassin
AppImage built with the contious build version from yesterday:
https://drive.google.com/open?id=1l8joVtBzrqmt_MWBIQVCb0X-Hqw2-h1z

[TheAssassin]

I don't trust nor like Google, perhaps use a non-Google host? (e.g., https://transfer.sh)

[sigmaboy]

I tried transfer.sh and the file is too big.
Error:
Due to abuse we've limited max upload size. Thank the pirates!

So is google at least okay for you? If not I don't know where I could upload files that big.
If you could give me another alternative I will upload it there too.

And I got an interesting info:
The testapp I created with the latest tools and linked above works well with a CentOS 7 x86_64 (current patch level).

I try the old one with CentOS aswell and report it to you.

[TheAssassin]

Didn't know that, must be a newer change. I'll get it from Google then, sorry.

[azubieta]

@probonopd can you help me migrating this issue to libappimage?

[azubieta]

Fixed at #20

[sigmaboy]

I dropped the issue because I had no more ideas left. @azubieta seemed to fix this but unfortunately this doesn't work for me. I still get the same error on the same filesystems. tmps, vfat and nfs still working, while the other don't. Could you maybe have another look into it. If you want, I can provide updated Appimage built with current versions of appimagetools.
@TheAssassin @probonopd Could you re-open the issue again?

[azubieta]

@sigmaboy Indeed I tried to solve the issue. Please make sure you are using the very latest versions of our tools. If this doesn't work feel free to share an updated AppIage for us to test.

[TheAssassin]

@azubieta is AppImageKit using the latest 0.1.x version of libappimage? I doubt it. Can you please update it there?

[azubieta]

@TheAssassin I don't have permissions to push to the AppImageKit master branch.

[TheAssassin]

There's PRs, you can update submodules there, too. But I'll quickly do it.

[sigmaboy]

That explained alot. But unfortunately it won't work with the latest version of libappimage.
I used the latest CI build (commit cddcc2e) which is the commit state TheAssassin made to update the git submodule.
Still the same error.
Here is a current AppImage build. It only calls a little "echo" bash script to verify it works.
https://drive.google.com/open?id=1dWXb7TJQTkrv4cyM7lb7nRKvq9YM9eoK

[sigmaboy]

@azubieta Could you maybe have another look at it?

[azubieta]

@sigmaboy sure, would you mind to ping me at freenode so we can talk about it and find a final solution.

[probonopd]

Can we make a test for this?

This is one of the errors we normally wouldn't catch quickly because the vast majority of AppImages is smaller than 200 MB. Hence, a test would be especially useful for such edge cases.

[TheAssassin]

@azubieta tested it in a "sane" way locally and thought the issue was fixed (i.e., in a way you would write such a system test, too). However, now it seems there was another edge case found where it failed again. A test wouldn't have captured that.

What I don't like about such a test is the high amount of file I/O which would be annoying for devs. It wears down SSDs quite quickly. Also, it seems to be quite tricky to create the environment to reproduce the bug, which is also a reason against such a test. And I'd never test such things on a totally unknown environment such as Travis CI -- who knows what they configured in their kernels etc.?

[sigmaboy]

@TheAssassin @azubieta
This is how I can reproduce the issue with a Ubuntu 18.10 Live Linux:

1. Download Ubuntu 18.10: http://cdimage.ubuntu.com/xubuntu/releases/18.10/release/xubuntu-18.10-desktop-amd64.iso
2. Boot the Xubuntu LIVE iso with KVM/libvirt with at least 4GB of RAM and internet access.
3. Install all dependencies to run the 32bit appimage (including sshd to access the live linux via ssh)

sudo dpkg --add-architecture i386
sudo apt-get update
sudo apt-get install libc6:i386 libncurses5:i386 libstdc++6:i386 libz1:i386 libfuse2:i386 openssh-server
sudo systemctl start sshd.service
sudo passwd xubuntu

Set the password as you wish
4. Download the following appimage to your client (not the vm): https://drive.google.com/file/d/1dWXb7TJQTkrv4cyM7lb7nRKvq9YM9eoK/view
5. Create an ext4 disk image

mkdir tmpfs
sudo mount -t tmpfs -o size=3G none tmpfs
dd if=/dev/zero of=./tmpfs/test.img bs=1M count=3072
mkfs.ext4 ./tmpfs/test.img
mkdir ext4
sudo mount ./tmpfs/test.img ext4
sudo chown xubuntu:xubuntu ./tmpfs/ext4/

6. Copy the appimage to your Live Linux with scp

client$ scp TestApp_2019-01-23.AppImage xubuntu@10.0.0.xxx:/home/xubuntu/ext4/

7. On the live Linux run the appimage

./ext4/TestApp_2019-01-23.AppImage

Alternative install some linux distribution with an ext or xfs filesystem and run the appimage from there.

As written in this comment. tmpfs, nfs and (v)fat are not affected by this bug

[probonopd]

Why would someone want to run a 32-bit AppImage on a 64-bit system? Seems like an odd edge case to me.

[TheAssassin]

Whenever there's just 32-bit binaries available that you want to (re-)package, and you don't want to ship 2 AppImages with the exact same contents.

[probonopd]

Well, even in those cases it may be advisable to make a 64-bit AppImage that contains both the 32-bit application, any 32-bit libraries it needs to run, and a 64-bit runtime, because those cannot reasonably assumed to be present on all 64-bit systems.

Personally I can live with a 2 GB limitation on 32-bit AppImages running on 64-bit systems and would not invest any time into this, as long as the 2 GB limitation is not there on 32-bit systems running 32-bit AppImages nor 64-bit systems running 64-bit AppImages (the latter is what we mainly care about those days).

[sigmaboy]

in some cases the 32bit payload in a 64 appimage won't work because of bugs(!?) inside the application. My goal was to build appimages for linux games for lan parties I host. And some of them crash when running from a 64bit appimage. Starting them regularly (without appimage) works perfectly fine with a 64bit OS. Games with the same engine smaller than 2GB work with a 32bit appimage.
This is the reason I'd like to have this issue fixed.

I haven't tested the appimage inside a 32bit linux. Atm I'm downloading 32bit xubuntu. But I think the problem exists also with 32bit linux distributions.
I will report @probonopd

/e: I tested and verified the bug with Xubuntu 18.04 i386
So basically everything regarding i386 architecture is affected. Not only 64bit linux with 32bit libraries

[probonopd]

OK, thanks for testing @sigmaboy

[TheAssassin]

So basically everything regarding i386 architecture is affected. Not only 64bit linux with 32bit libraries

I expected that, or rather, if it doesn't work in a 64-bit environment, it won't work in a 32-bit one for sure.

[sigmaboy]

Any news on that issue?

[probonopd]

As for me, I haven't investigated this: I don't have any 32-bit systems anymore, nor do I have any AppImage over 2 GB. Pull requests welcome though.

[darealshinji]

To be honest, for large video games I would just use something like linuxdeploy without creating an AppImage and then put the whole thing into a compressed archive. Or alternatively put only the game engine stuff into an AppImage and store the game files in a single PAK style format (like .vpk or .gcf files).