Nightly rhel8-arm64-release build failing

[richardlau]

Starting on 5 October, the rhel8-arm64-release nightly builds are failing,
e.g. https://ci-release.nodejs.org/job/iojs+release/11302/nodes=rhel8-arm64-release/

11:04:01   LD_LIBRARY_PATH=/home/iojs/build/ws/out/Release/lib.host:/home/iojs/build/ws/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../tools/v8_gypfiles; mkdir -p /home/iojs/build/ws/out/Release/obj.target/v8_snapshot/geni; "/home/iojs/build/ws/out/Release/mksnapshot" --turbo_instruction_scheduling --stress-turbo-late-spilling "--target_os=linux" "--target_arch=arm64" --startup_src "/home/iojs/build/ws/out/Release/obj.target/v8_snapshot/geni/snapshot.cc" --embedded_variant Default --embedded_src "/home/iojs/build/ws/out/Release/obj.target/v8_snapshot/geni/embedded.S" --concurrent-builtin-generation "--concurrent-turbofan-max-threads=0" --no-native-code-counters
11:04:01 
11:04:01 
11:04:01 #
11:04:01 # Fatal error in , line 0
11:04:01 # Check failed: 12 == (*__errno_location ()).
11:04:01 #
11:04:01 #
11:04:01 #
11:04:01 #FailureMessage Object: 0xffffcc1eecf0
11:04:01 ==== C stack trace ===============================
11:04:01 
11:04:01     /home/iojs/build/ws/out/Release/mksnapshot(v8::base::debug::StackTrace::StackTrace()+0x18) [0x359e340]
11:04:01     /home/iojs/build/ws/out/Release/mksnapshot() [0x1e56874]
11:04:01     /home/iojs/build/ws/out/Release/mksnapshot(V8_Fatal(char const*, ...)+0x1d8) [0x1e48f88]
11:04:01     /home/iojs/build/ws/out/Release/mksnapshot(v8::base::OS::SetPermissions(void*, unsigned long, v8::base::OS::MemoryPermission)+0x94) [0x1e4fbc0]
11:04:01     /home/iojs/build/ws/out/Release/mksnapshot(v8::base::VirtualAddressSubspace::AllocatePages(unsigned long, unsigned long, unsigned long, v8::PagePermissions)+0x64) [0x1e4f184]
11:04:01     /home/iojs/build/ws/out/Release/mksnapshot(v8::internal::ExternalEntityTable<v8::internal::JSDispatchEntry, 268435456ul>::Initialize()+0x9c) [0x15654a8]
11:04:01     /home/iojs/build/ws/out/Release/mksnapshot(v8::internal::IsolateGroup::InitializeOncePerProcess()+0x90) [0x15655e4]
11:04:01     /home/iojs/build/ws/out/Release/mksnapshot(v8::internal::V8::Initialize()+0x1d4) [0x1566790]
11:04:01     /home/iojs/build/ws/out/Release/mksnapshot(v8::V8::Initialize(int)+0x20) [0x1321650]
11:04:01     /home/iojs/build/ws/out/Release/mksnapshot(main+0x8cc) [0x130549c]
11:04:01     /lib64/libc.so.6(__libc_start_main+0xdc) [0xffffa6384384]
11:04:01     /home/iojs/build/ws/out/Release/mksnapshot() [0x1302410]
11:04:01 /bin/sh: line 1: 2604538 Trace/breakpoint trap   (core dumped) "/home/iojs/build/ws/out/Release/mksnapshot" --turbo_instruction_scheduling --stress-turbo-late-spilling "--target_os=linux" "--target_arch=arm64" --startup_src "/home/iojs/build/ws/out/Release/obj.target/v8_snapshot/geni/snapshot.cc" --embedded_variant Default --embedded_src "/home/iojs/build/ws/out/Release/obj.target/v8_snapshot/geni/embedded.S" --concurrent-builtin-generation "--concurrent-turbofan-max-threads=0" --no-native-code-counters
11:04:01 make[2]: *** [tools/v8_gypfiles/v8_snapshot.target.mk:17: 9ea847fdb69e2e49207cd1a7d9992a6b592d15fd.intermediate] Error 133
11:04:01 rm 97b3b35c0261bedd03bbbab54f6b138f20a11b4f.intermediate 0d2ce10ea44d84275198a857d5f920d0af8bab0c.intermediate 9ea847fdb69e2e49207cd1a7d9992a6b592d15fd.intermediate d8e3a47e1673322d324ac597a559f10bdc575bdf.intermediate
11:04:01 make[1]: *** [Makefile:135: node] Error 2
11:04:01 make: *** [Makefile:1316: node-v25.0.0-nightly2025100532851f31c2-linux-arm64.tar] Error 2

This only appears to be affecting the nightly build from main -- the V8 nightly builds are still passing for rhel8-arm64-release.

[richardlau]

https://ci-release.nodejs.org/job/iojs+release/nodes=rhel8-arm64-release/11300/ passed on nodejs/node@6f941fc.
https://ci-release.nodejs.org/job/iojs+release/nodes=rhel8-arm64-release/11302/ failed on nodejs/node@32851f3.

nodejs/node@6f941fc...32851f3 includes the V8 14.1 update.

[richardlau]

AFAIK we're not seen anything like this in the test CI.

[richardlau]

I updated and rebooted release-osuosl-rhel8-arm64-1 but problem persists: https://ci-release.nodejs.org/job/iojs+release/nodes=rhel8-arm64-release/11310/console

I might try wiping out the ccache in case it is a cache issue, but I might wait for the release build for 24.10.0 as wiping out the cache will make builds take longer.

[richardlau]

I've wiped out the ccache cache and rerunning today's (still failing) nightly:
https://ci-release.nodejs.org/job/iojs+release/11314/nodes=rhel8-arm64-release/

[richardlau]

I've wiped out the ccache cache and rerunning today's (still failing) nightly: https://ci-release.nodejs.org/job/iojs+release/11314/nodes=rhel8-arm64-release/

Clearing the ccache hasn't made a difference and mksnapshot is still failing a check. This doesn't fail in the test CI and AFAICT we're on the same compiler (clang 19). I'm going to need help to progress this further.

[richardlau]

Questions:

• Why does this happen on main in the release CI but not for v8-canary from https://github.com/nodejs/node-v8.git?
• Why is this occurring on the release machine (rhel8-arm64 at OSUOSL) but not on the test CI (passing on both the Azure and OSUOUSL machines)?
• What has changed? Is it actually related to the V8 14.1 update? I didn't spot anything obvious in the commit range in Nightly rhel8-arm64-release build failing #4172 (comment).

I've just found nodejs/node#55509, which is a similar check failure from v8::base::OS::SetPermissions albeit that was reported on x64 and this is arm64.

[richardlau]

If we're in v8::base::OS::SetPermissions I wonder if this is failing the CHECK_EQ at https://github.com/nodejs/node/blob/75a6fff6befa00dd4338bca5dd260a7478989965/deps/v8/src/base/platform/platform-posix.cc#L557?

  // Setting permissions can fail if the limit of VMAs is exceeded.
  // Any failure that's not OOM likely indicates a bug in the caller (e.g.
  // using an invalid mapping) so attempt to catch that here to facilitate
  // debugging of these failures.
  if (ret != 0) CHECK_EQ(ENOMEM, errno);

at least ENOMEM is 12 on this platform (not sure how errno would map to (*__errno_location ())). Maybe this is memory related (I think the release machine has less available memory than the test machines)?

[richardlau]

FWIW

Instance Name	VCPUs	Disk	RAM
test-osuosl-ubuntu2004_docker-arm64-1	16	320GB	32GB
release-osuosl-rhel8-arm64-1	2	30GB	4GB

Machine has 4GB of swap:

[iojs@release-osuosl-rhel8-arm64-1 ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:          3.4Gi       459Mi       831Mi        43Mi       2.1Gi       2.4Gi
Swap:         4.0Gi       152Mi       3.9Gi
[iojs@release-osuosl-rhel8-arm64-1 ~]$

[richardlau]

There's also nodejs/help#4465, which includes strace commands to run for further debugging. There's references there to a MemoryDenyWriteExecute setting in the systemd file in that issue but I checked the file for jenkins on the machine and didn't find any equivalent set.

[richardlau]

It might be https://issues.chromium.org/issues/425634685.

[richardlau]

It might be https://issues.chromium.org/issues/425634685.

So this looks plausible and has supporting circumstantial evidence.

Going back further in history:

• node-v8 canary branch build on 22 September failed in the same way .
• The next canary build on 24 September and onwards passed.
• Earlier builds for canary back to 13 September (earliest I can see history for) failed in the same way.

v8/v8@1e190bb landed upstream V8 on 23 September. It's not in the version of V8 we currently have on main.

I'm currently rebuilding on the release machine and planning to see if applying v8/v8@1e190bb fixes the failure. I'm not going to be around much over the weekend, so if the rebuild doesn't complete in a timely manner today I might just open the backport PR.

[targos]

Amazing investigation!