strange CI error running gc code

[vtjnash]

This seems it clearly shouldn't happen, as the Matrix is thread-local and the size is not changed, so there is not much way the length of the Memory inside should be corrupted, but yet somehow, we see this failure:

gc                                                (1) |        started at 2024-01-21T00:19:42.883
ERROR: LoadError: TaskFailedException
    nested task error: BoundsError: attempt to access 1125×1125 Matrix{Cell} at index [1]
    Stacktrace:
     [1] throw_boundserror(A::Matrix{Cell}, I::Tuple{Int64})
       @ Base ./essentials.jl:14
     [2] setindex!
       @ ./array.jl:971 [inlined]
     [3] fillcells!(mc::Matrix{Cell})
       @ Main /cache/build/tester-amdci5-12/julialang/julia-master/julia-c5e9e7c283/share/julia/test/gc/objarray.jl:18
     [4] work
       @ /cache/build/tester-amdci5-12/julialang/julia-master/julia-c5e9e7c283/share/julia/test/gc/objarray.jl:25 [inlined]
     [5] macro expansion
       @ /cache/build/tester-amdci5-12/julialang/julia-master/julia-c5e9e7c283/share/julia/test/gc/objarray.jl:30 [inlined]
     [6] (::var"#7#threadsfor_fun#2"{var"#7#threadsfor_fun#1#3"{UnitRange{Int64}}})(tid::Int64; onethread::Bool)
       @ Main ./threadingconstructs.jl:214
     [7] #7#threadsfor_fun
       @ ./threadingconstructs.jl:181 [inlined]
     [8] (::Base.Threads.var"#1#2"{var"#7#threadsfor_fun#2"{var"#7#threadsfor_fun#1#3"{UnitRange{Int64}}}, Int64})()
       @ Base.Threads ./threadingconstructs.jl:153
Stacktrace:
 [1] threading_run(fun::var"#7#threadsfor_fun#2"{var"#7#threadsfor_fun#1#3"{UnitRange{Int64}}}, static::Bool)
   @ Base.Threads ./threadingconstructs.jl:171
 [2] macro expansion
   @ ./threadingconstructs.jl:219 [inlined]
 [3] run(maxsize::Int64)
   @ Main /cache/build/tester-amdci5-12/julialang/julia-master/julia-c5e9e7c283/share/julia/test/gc/objarray.jl:29
 [4] top-level scope
   @ /cache/build/tester-amdci5-12/julialang/julia-master/julia-c5e9e7c283/share/julia/test/gc/objarray.jl:35
in expression starting at /cache/build/tester-amdci5-12/julialang/julia-master/julia-c5e9e7c283/share/julia/test/gc/objarray.jl:35
gc                                                (1) |         failed at 2024-01-21T00:20:30.964

https://buildkite.com/julialang/julia-master/builds/32516#018d2921-a653-440d-aefb-55eb83d5595c

[vtjnash]

More apparent examples of GC issues with this test

gc                                                (1) |        started at 2024-02-01T14:28:27.010
ERROR: LoadError: TaskFailedException
    nested task error: StackOverflowError:
    Stacktrace:
     [1] check(node::Main.BinaryTreeMutable.Node)
       @ Main.BinaryTreeMutable /cache/build/tester-amdci5-12/julialang/julia-master/julia-30ccace427/share/julia/test/gc/binarytree.jl:20
     [2] check(node::Main.BinaryTreeMutable.Node) (repeats 32246 times)
       @ Main.BinaryTreeMutable /cache/build/tester-amdci5-12/julialang/julia-master/julia-30ccace427/share/julia/test/gc/binarytree.jl:21
     [3] macro expansion
       @ /cache/build/tester-amdci5-12/julialang/julia-master/julia-30ccace427/share/julia/test/gc/binarytree.jl:35 [inlined]
     [4] (::Main.BinaryTreeMutable.var"#5#threadsfor_fun#2"{Main.BinaryTreeMutable.var"#5#threadsfor_fun#1#3"{Int32, Vector{String}, Int32, StepRange{Int32, Int32}}})(tid::Int32; onethread::Bool)
       @ Main.BinaryTreeMutable ./threadingconstructs.jl:215
     [5] #5#threadsfor_fun
       @ ./threadingconstructs.jl:182 [inlined]
     [6] (::Base.Threads.var"#1#2"{Main.BinaryTreeMutable.var"#5#threadsfor_fun#2"{Main.BinaryTreeMutable.var"#5#threadsfor_fun#1#3"{Int32, Vector{String}, Int32, StepRange{Int32, Int32}}}, Int32})()
       @ Base.Threads ./threadingconstructs.jl:154
Stacktrace:
 [1] threading_run(fun::Main.BinaryTreeMutable.var"#5#threadsfor_fun#2"{Main.BinaryTreeMutable.var"#5#threadsfor_fun#1#3"{Int32, Vector{String}, Int32, StepRange{Int32, Int32}}}, static::Bool)
   @ Base.Threads ./threadingconstructs.jl:172
 [2] macro expansion
   @ ./threadingconstructs.jl:220 [inlined]
 [3] binary_trees(io::Base.DevNull, n::Int32)
   @ Main.BinaryTreeMutable /cache/build/tester-amdci5-12/julialang/julia-master/julia-30ccace427/share/julia/test/gc/binarytree.jl:31
 [4] top-level scope
   @ /cache/build/tester-amdci5-12/julialang/julia-master/julia-30ccace427/share/julia/test/gc/binarytree.jl:53
in expression starting at /cache/build/tester-amdci5-12/julialang/julia-master/julia-30ccace427/share/julia/test/gc/binarytree.jl:53
GC error (probable corruption)
Allocations: 1790315 (Pool: 1790283; Big: 32); GC: 7
!!! ERROR in jl_ -- ABORTING !!!
[8571] signal 6 (-6): Aborted
in expression starting at /cache/build/tester-amdci5-12/julialang/julia-master/julia-30ccace427/share/julia/test/gc/objarray.jl:35
Allocations: 1790315 (Pool: 1790283; Big: 32); GC: 7
gc                                                (1) |         failed at 2024-02-01T14:29:44.974
Error During Test at /cache/build/tester-amdci5-12/julialang/julia-master/julia-30ccace427/share/julia/test/gc.jl:12
  Test threw exception

https://buildkite.com/julialang/julia-master/builds/33000#018d64bc-df85-41f7-bf7d-cfa160941109
@d-netto

[d-netto]

I started doing a bisect on this, but cycle time is fairly slow since this takes a while to reproduce in the GC tests.

I plan to post an update here when I'm done bisecting.

[vtjnash]

If helpful, there is also an rr trace of this in #52757

[d-netto]

Unable to reproduce it after leaving it running for a few hundred times in a loop... Will look at the rr trace.