Allocating ridiculous amounts of memory hangs in GC

[maleadt]

julia> Base.format_bytes(17314512175606968824)
"15378.376 PiB"

julia> ccall(:jl_gc_counted_malloc, Ptr{Cvoid}, (Csize_t,), 17314512175606968824)
# hangs

Interrupting the process shows that it seems to be stuck doing GC, which doesn't make sense here. Maybe an issue with the GC counters?

[1534036] signal (15): Terminated
in expression starting at none:0
gc_mark_obj8 at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gc.c:1880
gc_mark_outrefs at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gc.c:2660 [inlined]
gc_mark_loop_serial_ at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gc.c:2716
gc_mark_loop_serial at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gc.c:2739
gc_mark_loop at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gc.c:2852 [inlined]
_jl_gc_collect at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gc.c:3178
ijl_gc_collect at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gc.c:3478
maybe_collect at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gc.c:941 [inlined]
jl_gc_pool_alloc_inner at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gc.c:1320
jl_gc_pool_alloc_noinline at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gc.c:1377 [inlined]
jl_gc_alloc_ at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia_internal.h:466 [inlined]
jl_gc_alloc at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gc.c:3530
_new_array_ at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/array.c:134 [inlined]
_new_array at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/array.c:198 [inlined]
ijl_alloc_array_1d at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/array.c:436

There also seems to be a difference between doing this from the REPL, and via --eval:

❯ ./julia -e '@show ccall(:jl_gc_counted_malloc, Ptr{Cvoid}, (Csize_t,), 17314512175606968824)'
ccall(:jl_gc_counted_malloc, Ptr{Cvoid}, (Csize_t,), 17314512175606968824) = Ptr{Nothing} @0x0000000000000000

Bisected to 8cfb350 on the backports branch, so #50682 is probably the culprit.

[gbaraldi]

I believe the issue is that we don't check if we actually got an allocation or not before increasing the counters, and the GC will then think it has a massive amount of memory allocated and will run very often. Though i'm not sure what can we do here, because a null check might not be good enough because the OS might give us a pointer and then segfault later.

[maleadt]

a null check might not be good enough because the OS might give us a pointer and then segfault later

Could you elaborate? We should at least handle the NULL case.

[oscardssmith]

The problem is that the OS might not actually give us the pages until we touch them so it's possible for the maloc to succeed but for us to get a segfault once we actually try to fill it with data.

[maleadt]

I know how memory overcommit works, but I haven't seen it segfault, only OOM kills.

In any case, this doesn't matter here, as malloc actually returns NULL here. We used to handle this fine, and don't anymore, which breaks GMP.

[gbaraldi]

I guess it won't segfault, but it will fail. In any case I have a PR that at least checks null l.