Error/Segfault when writing many partitions

[svilupp]

When writing a partitioned table with compression, it throws an error or segfaults.
It impacts Julia 1.8.5 on both ARM and Linux when threading is enabled. Possible root causes are suggested below.

How to reproduce:

using Arrow, Tables, Random
using Logging
debuglogger = ConsoleLogger(stderr, Logging.Debug)

# Create dataset
N = 64 # set to 2x Threads.nthreads()
len = 100
t = Tables.rowtable((; x1 = map(x -> randstring(len), 1:N)))

# This runs okay
Arrow.write("test_partitioned.arrow", Iterators.partition(t, 1); compress = :zstd);

# This fails with Threads.nthreads()>1 and compression enabled (logging causes a yield, which creates the collision
with_logger(debuglogger) do
    # fails for both :zstd and :lz4
    Arrow.write("test_partitioned.arrow", Iterators.partition(t, 1); compress = :lz4)
end
# if it doesn't fail immediately, run it a few times

If you don't use compression, have 1 thread, or don't use logging, or don't use partitioned data (to enabled threaded workload) all is fine.

On Linux, it results in the following errors:

# ERROR: fatal error writing arrow data
# or: InvalidStateException: Channel is closed. ERROR: fatal error writing arrow data

on ARM (Apple M1), it either gives the same errors, or sometimes it segfaults:

# signal (11): Segmentation fault: 11
# in expression starting at /Users/x/arrow-internals/bug_with_partitions.jl:15
# jl_table_peek_bp at /Users/x/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
# ijl_eqtable_get at /Users/x/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)

# signal (11): Segmentation fault: 11
# in expression starting at /Users/x/arrow-internals/bug_with_partitions.jl:15

# signal (11): Segmentation fault: 11
# in expression starting at /Users/x/arrow-internals/bug_with_partitions.jl:15
# _ZN5dyld44APIs6dladdrEPKvP7dl_info at /usr/lib/dyld (unknown line)
# Allocations: 103975646 (Pool: 103867433; Big: 108213); GC: 82

# signal (11): Segmentation fault: 11
# in expression starting at /Users/x/arrow-internals/bug_with_partitions.jl:15
# subtype at /Users/x/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
# forall_exists_subtype at /Users/x/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
# ijl_subtype_env at /Users/x/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
# jl_type_intersection_env_s at /Users/x/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)

# signal (11): Segmentation fault: 11
# in expression starting at /Users/x/arrow-internals/bug_with_partitions.jl:15
# _ZN5dyld44APIs6dladdrEPKvP7dl_info at /usr/lib/dyld (unknown line)
# Allocations: 103975646 (Pool: 103867433; Big: 108213); GC: 82

Suspected root cause:

• This line where the list of pre-initialized compressors are accessed by Threads.threadid()
• From Julia 1.8, :dynamic thread scheduler ships as a default, so if any part of the code yields (for example when using logging - vis. above!), different tasks might try to tap into the same compressor
• See the issue discussed on Floops.jl and here

Versioninfo:

Arrow 2.4.3 (both the released and on main branch)

Julia Version 1.8.5
Commit 17cfb8e65ea (2023-01-08 06:45 UTC)
Platform Info:
OS: Linux (x86_64-linux-gnu)
CPU: 32 × Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-13.0.1 (ORCJIT, cascadelake)
Threads: 1 on 32 virtual cores

Julia Version 1.8.5
Commit 17cfb8e65ea (2023-01-08 06:45 UTC)
Platform Info:
OS: macOS (arm64-apple-darwin21.5.0)
CPU: 8 × Apple M1 Pro
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
Threads: 1 on 6 virtual cores