linux-arm and linux-arm64 prebuild comes with GLIBC which is too new for Raspberry Pi

[cswendrowski]

The readme lists that this should work on Raspberry Pi, but we've traced down that due to a GLIBC issue, this is not the case

Reproduced on an ARM64 laptop running Ubuntu 18:

Error: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/cswendrowski/foundryvtt/resources/app/node_modules/classic-level/prebuilds/linux-arm64/node.napi.armv8.node)

Ubuntu 18 ships with GLIBC 2.27, Raspian and Pi OS 10 ships 2.28, and Pi OS 11 with 2.29

On an x64 machine also running Ubuntu 18, the prebundle comes compiled with an older version of GLIBC and runs correctly on 2.27

Any guidance or assistance would be appreciated

[vweevers]

We build using dockcross docker images that are unversioned and we don't actively test Raspberry Pi, so it's very well possible that we inadvertently broke it.

I don't have time to look into it. Could you check the glibc versions of older prebuilds (maybe even going back to leveldown)? And/or check what changed in dockcross, if anything. For context, we use prebuildify-cross to build the binaries.

Note: at some point we do have to end support of some platforms (intentionally) because node 18 moved to glibc 2.28.

[cswendrowski]

It looks like there's at least one temporary easy win, although it will pose a problem for Node 18 - prebuild/docker-images for prebuildify-cross offers linux-arm64-lts, which uses glibc 2.27 for compatibility: https://github.com/prebuild/docker-images/pkgs/container/linux-arm64-lts

Dockcross additionally has the following that should meet the bill (https://github.com/dockcross/dockcross#summary-cross-compilers):

• dockcross/linux-armv6-lts GLIBC 2.28
• dockcross/linux-armv7-lts GLIBC 2.28

They don't have a prebuildify wrapped verison, however, so changes may need to be submitted to them.

I'll try to figure out getting these builds setup on my machine and start seeing if there's an option to go to an older prebuild, although the right longterm answer initially seems to be getting prebuilds of the above Dockcross images

Note: at some point we do have to end support of some platforms (intentionally) because node 18 moved to glibc 2.28.

I think that's fair (and necessary). Ubuntu 18's 2.27 was just an easy target to test with to debug this issue

[vweevers]

prebuild/docker-images for prebuildify-cross offers linux-arm64-lts

Ah right, I forgot (though I was the one who released that).

dockcross/linux-armv6-lts
dockcross/linux-armv7-lts

PRs are welcome over at prebuild/docker-images! I won't be able to review in depth but that's OK seeing as it would be new images (rather than changes to existing images that are in active use).

[cswendrowski]

Hey there @vweevers, I chatted with my team and we'd like to invest some effort into solving this problem, but I wanted to get your input into how to best accomplish the work and verify the changes work.

Step 1) It seems pretty straightforward to update https://github.com/Level/classic-level/blob/main/package.json#L21 to use linux-arm64-lts. I can do this locally and build on my ARM64 laptop to verify classic level runs as expected with that option, then open a PR. I don't think this requires any updates to prebuildify-cross, but let me know if I'm wrong

Step 2) Looking at the changes between https://github.com/prebuild/docker-images/tree/master/linux-arm64 and https://github.com/prebuild/docker-images/tree/master/linux-arm64-lts, it seems like the only thing I'd need to do would be to make copies of the armv6 and armv7 images and have them inherit from the dockcross images identified. What I'm unsure on is how to verify that the changes are correct, and whether prebuildify-cross needs an update to see the new images. Any insights would be helpful

[vweevers]

Step 1

That's correct.

Step 2

To verify, build the new docker image(s) locally, then create a container with the classic-level repo mounted as a volume (or clone the git repo inside the container, but then npm install might not work for lack of root IIRC). Then run prebuildify -t 8.14.0 --napi --strip in the container (with classic-level as working directory). Which should give you a functional prebuild.

Note that prebuildify-cross merely automates these steps. In theory it also supports local docker images but reportedly that's broken (prebuild/prebuildify-cross#16).

Lastly, prebuildify-cross does not need an update, because it defaults to pulling images from ghcr.io/prebuild/<image> where <image> is an image name provided via the -i command line flag.

I appreciate your help!

[cswendrowski]

@vweevers I've been running into some issues getting a build made via prebuildify-cross:

gyp: name 'openssl_fips' is not defined while evaluating condition 'openssl_fips != ""' in /home/node/app/deps/leveldb/leveldb.gyp while loading dependencies of binding.gyp while trying to load binding.gyp

I've tried manually setting openssl_fips as an environment variable, manually setting it as a variable in binding.gyp, and updating my libssl-dev without any success

I even tried via a fork of the current Github action, which failed with the same error: https://github.com/cswendrowski/classic-level/actions/runs/4441111255/jobs/7795775652

Unfortunately the logs of the last successful run on this repo are no longer retained (https://github.com/Level/classic-level/actions/runs/2039662848/jobs/2944228919), so I can't see how it succeeded previously.

Any pointers would be appreciated - as a Windows dev, I don't normally dig into the world of compiling C++ on Linux, but I'm trying my best

EDIT: Well, as always, asking questions is a great way to debug. I was able to get past this issue by adding the following block to binding.gyp, leveldb.gyp, and snappy.gyp:

"variables": {
  'openssl_fips': ''
}

This doesn't feel like the right answer, and if it is, we'll still need to get those updated in the upstream repos since two of these are git submodule files, but it at least got a build done on my machine

[vweevers]

Doesn't ring a bell, I'll check it later and/or trigger a GHA build here.

[cswendrowski]

Back again with an update after much research:

I was able to wrap the dockcross/linux-armv6-lts and get a build made, but it turns out that Node 16 is not available on armv6, nor does it seem it ever will be. I can still open a PR for this Dockerfile, but it seems like armv6 support should be dropped.

I was unable to do a successful build using a wrapped dockcross/linux-armv7-lts due to the following issue: dockcross/dockcross#718
Per https://en.wikipedia.org/wiki/Raspberry_Pi#Specifications, the only Pi that uses armv7 is the Pi 2 Model B v1.1, which we are content to drop support for ourselves. I've left a comment on that dockcross issue in hopes that they one day resolve that issue, but I was unable to solve it myself.

That leaves arm64, which we have successfully verified on newer Pi Models. The only work to support those targets is this PR: #71

Thanks for your support in debugging this issue, and we look forward to consuming this change when available