[CI] aarch64-linux-gnu test job is systematically hanging

[giordano]

I haven't found when this started (2f5e1d0 is the earliest commit where I've seen this, but haven't looked further back), but aarch64-linux-gnu jobs are systematically hanging, for example: https://buildkite.com/julialang/julia-master/builds/50445#0199318a-6ad9-4f3d-b7a2-ad4e0191cb97/933-1634

errorshow                                        (8) |            has been running for 1h 0m           
channels                                        (10) |            has been running for 1h 0m           
subarray                                         (9) |            has been running for 1h 15m          
SparseArrays/cholmod                             (5) |             has been running for 45m            
read                                             (6) |            has been running for 1h 15m          
broadcast                                        (4) |            has been running for 1h 15m          
errorshow                                        (8) |            has been running for 1h 15m          
channels                                        (10) |            has been running for 1h 15m          
subarray                                         (9) |            has been running for 1h 30m          
SparseArrays/cholmod                             (5) |            has been running for 1h 0m           
read                                             (6) |            has been running for 1h 30m          
broadcast                                        (4) |            has been running for 1h 30m          

Testsets like read and errorshow should take few seconds, not well over one hour (this is on an x86_64-linux-gnu machine):

julia> Base.runtests(["read", "errorshow"])
Running parallel tests with:
  getpid() = 109433
  nworkers() = 2
  nthreads(:interactive) = 1
  nthreads(:default) = 1
  Sys.CPU_THREADS = 11
  Sys.total_memory() = 30.809 GiB
  Sys.free_memory() = 20.524 GiB

Test  (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB)
read       (2) |        started at 2025-09-12T20:00:30.647
errorshow  (3) |        started at 2025-09-12T20:00:30.728
      From worker 3:    Warning: detected a stack overflow; program state may be corrupted, so further execution might be unreliable.
      From worker 3:    Warning: detected a stack overflow; program state may be corrupted, so further execution might be unreliable.
errorshow  (3) |     9.71 |   0.24 |  2.5 |    1108.73 |   510.00
read       (2) |    16.57 |   0.74 |  4.5 |    3047.27 |  7885.12

Test Summary: | Pass  Total   Time
  Overall     | 7080   7080  19.1s

[giordano]

Weirdly enough, first commit on master where this started hanging seems to be 5c62500 (CI run), after that all jobs timed out after 2 hours, before that test jobs finished in ~90 minutes.

I need to double check, #54839 might be have been fixed by #59493 (update of libunwind to v1.8.3, whose release notes mentions a fix of an off-by-one error on aarch64), I haven't spotted new errors like that after #59493 was merged, but with all jobs systematically failing it's more complicated to verify that. Edit: nevermind, I can still reproduce locally the test failure in Profile on 9427f33.

[vtjnash]

Duplicate of #59538

[giordano]

What's the evidence for that?

[DilumAluthge]

(I'm not saying I agree or disagree - just that if it is a duplicate, we should close it as "gray", not "purple".)

[giordano]

The working hypothesis is that this is a manifestation of the known issue with jitlink using too many memory mappings: apparently around August a new set of CI machines started picking up the aarch64-linux jobs, but they didn't have the workaround suggested in the docs applied. @Keno just changed it noticing the error message in #59658, and https://buildkite.com/julialang/julia-master/builds/50822#01998245-4453-49ea-bde8-3d0ab2014ee6, first job since the workaround was applied to the new CI machines, doesn't seem to be hanging. We'll see how it goes, but it's hopefully only an infrastructure issue (besides the upstream llvm issue).

[giordano]

After the infrastructure change to the CI machine as part of #59658 we didn't have any test aarch64-linux-gnu job timing out between https://buildkite.com/julialang/julia-master/builds/50822 and https://buildkite.com/julialang/julia-master/builds/50840 (however some builds were cancelled before the test jobs even started), I'd say that's enough of an evidence that this is indeed a problem only with the CI machine, all previous jobs systematically timed out.