Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segmentation Fault with Threads.@spawn + Tables.partitioner + write with compression #82

Closed
altre opened this issue Dec 22, 2020 · 4 comments · Fixed by #118
Closed

Segmentation Fault with Threads.@spawn + Tables.partitioner + write with compression #82

altre opened this issue Dec 22, 2020 · 4 comments · Fixed by #118

Comments

@altre
Copy link

altre commented Dec 22, 2020

I'm not completely certain where the problem actually is here, but this works when removing compression or not using the partitioner:

using Arrow
using DataFrames

const flup = DataFrame([[rand('a':'h') for _ in 1:300] for _ in 'a':'h'], Symbol.('a':'h'))
seqis = [Iterators.partition(1:300, 20) for _ in 1:100]
d = mktempdir()
try 
    t = Threads.@spawn for (i,is) in enumerate(seqis)
        Arrow.write(joinpath(d, "$i.arrow"), Tables.partitioner((flup[i,:] for i in is)); compress=:lz4)
    end
    wait(t)
finally
    rm(d, recursive=true)
end

Causes:

compress at /home/ec2-user/.julia/packages/Arrow/CyJ4L/src/arraytypes/primitive.jl:77
unknown function (ip: 0x7f157f2f25ff)
jl_apply_generic at /usr/bin/../lib64/libjulia.so.1 (unknown line)
double free or corruption (out)
#toarrowvector#3 at /home/ec2-user/.julia/packages/Arrow/CyJ4L/src/arraytypes/arraytypes.jl:38

signal (6): Aborted
in expression starting at none:0
toarrowvector##kw at /home/ec2-user/.julia/packages/Arrow/CyJ4L/src/arraytypes/arraytypes.jl:34 [inlined]
toarrowvector##kw at /home/ec2-user/.julia/packages/Arrow/CyJ4L/src/arraytypes/arraytypes.jl:34 [inlined]
#113 at /home/ec2-user/.julia/pac#113 at /home/ec2-user/.julia/packages/Arrow/CyJ4L/src/write.jl:202
unknown function (ip: 0x7f157f2e9unknown function (ip: 0x7f157f2e904e)

signal (11): Segmentation fault
in expression starting at none:0   
LZ4HC_InsertAndGetWiderMatch.constprop.5 at /home/ec2-user/.julia/artifacts/25d411cb0031c3b42c10c945cdf6eb5253cb44b7/lib/liblz4.so (unknown line)
LZ4HC_compress_generic_internal.part.1 at /home/ec2-user/.julia/artifacts/25d411cb0031c3b42c10c945cdf6eb5253cb44b7/lib/liblz4.so (unknown line)
LZ4_compress_HC_continue at /home/ec2-user/.julia/artifacts/25d411cb0031c3b42c10c945cdf6eb5253cb44b7/lib/liblz4.so (unknown line)
LZ4F_flush at /home/ec2-user/.julia/artifacts/25d411cb0031c3b42c10c945cdf6eb5253cb44b7/lib/liblz4.so (unknown line)
LZ4F_compressEnd at /home/ec2-user/.julia/artifacts/25d411cb0031c3b42c10c945cdf6eb5253cb44b7/lib/liblz4.so (unknown line)
LZ4F_compressEnd at /home/ec2-user/.julia/packages/CodecLz4/2JFgC/src/headers/lz4frame.jl:297
process at /home/ec2-user/.julia/packages/CodecLz4/2JFgC/src/frame_compression.jl:129
transcode at /home/ec2-user/.julia/packages/TranscodingStreams/MsN8d/src/transcode.jl:90
compress at /home/ec2-user/.julia/packages/Arrow/CyJ4L/src/arraytypes/compressed.jl:45 [inlined]
compress at /home/ec2-user/.julia/packages/Arrow/CyJ4L/src/arraytypes/compressed.jl:49 [inlined]
signal (11): Segmentation fault
in expression starting at none:0
jl_apply_generic at /usr/bin/../lib64/libjulia.so.1 (unknown line)
eachcolumn at /home/ec2-user/.julia/packages/Tables/iG2a3/src/utils.jl:70
unknown function (ip: 0x7f157f2e748c)
jl_apply_generic at /usr/bin/../lib64/libjulia.so.1 (unknown line)
toarrowtable at /home/ec2-user/.julia/packages/Arrow/CyJ4L/src/write.jl:201
unknown function (ip: 0x7f157f2e5d76)
jl_apply_generic at /usr/bin/../lib64/libjulia.so.1 (unknown line)
macro expansion at /home/ec2-user/.julia/packages/Arrow/CyJ4L/src/write.jl:127 [inlined]
#109 at ./threadingconstructs.jl:169
unknown function (ip: 0x7f157f2fa13c)
gsignal at /usr/bin/../lib64/libc.so.6 (unknown line)
jl_apply_generic at /usr/bin/../lib64/libjulia.so.1 (unknown line)
abort at /usr/bin/../lib64/libc.so.6 (unknown line)
unknown function (ip: 0x7f15bb6b95dd)
unknown function (ip: (nil))
Allocations: 48301139 (Pool: 48286788; Big: 14351); GC: 51
LZ4HC_InsertAndGetWiderMatch.constprop.5 at /home/ec2-user/.julia/artifacts/25d411cb0031c3b42c10c945cdf6eb5253cb44b7/lib/liblz4.so (unknown line)
LZ4HC_compress_generic_internal.part.1 at /home/ec2-user/.julia/artifacts/25d411cb0031c3b42c10c945cdf6eb5253cb44b7/lib/liblz4.so (unknown line)
LZ4_compress_HC_continue at /home/ec2-user/.julia/artifacts/25d411cb0031c3b42c10c945cdf6eb5253cb44b7/lib/liblz4.so (unknown line)
LZ4F_flush at /home/ec2-user/.julia/artifacts/25d411cb0031c3b42c10c945cdf6eb5253cb44b7/lib/liblz4.so (unknown line)
LZ4F_compressEnd at /home/ec2-user/.julia/artifacts/25d411cb0031c3b42c10c945cdf6eb5253cb44b7/lib/liblz4.so (unknown line)
LZ4F_compressEnd at /home/ec2-user/.julia/packages/CodecLz4/2JFgC/src/headers/lz4frame.jl:297
process at /home/ec2-user/.julia/packages/CodecLz4/2JFgC/src/frame_compression.jl:129
transcode at /home/ec2-user/.julia/packages/TranscodingStreams/MsN8d/src/transcode.jl:90
compress at /home/ec2-user/.julia/packages/Arrow/CyJ4L/src/arraytypes/compressed.jl:45 [inlined]
compress at /home/ec2-user/.julia/packages/Arrow/CyJ4L/src/arraytypes/compressed.jl:49 [inlined]
compress at /home/ec2-user/.julia/packages/Arrow/CyJ4L/src/arraytypes/primitive.jl:77
unknown function (ip: 0x7f157f2f25ff)
jl_apply_generic at /usr/bin/../lib64/libjulia.so.1 (unknown line)
#toarrowvector#3 at /home/ec2-user/.julia/packages/Arrow/CyJ4L/src/arraytypes/arraytypes.jl:38
toarrowvector##kw at /home/ec2-user/.julia/packages/Arrow/CyJ4L/src/arraytypes/arraytypes.jl:34 [inlined]
toarrowvector##kw at /home/ec2-user/.julia/packages/Arrow/CyJ4L/src/arraytypes/arraytypes.jl:34 [inlined]
#113 at /home/ec2-user/.julia/packages/Arrow/CyJ4L/src/write.jl:202
unknown function (ip: 0x7f157f2e904e)
jl_apply_generic at /usr/bin/../lib64/libjulia.so.1 (unknown line)
eachcolumn at /home/ec2-user/.julia/packages/Tables/iG2a3/src/utils.jl:70
unknown function (ip: 0x7f157f2e748c)
jl_apply_generic at /usr/bin/../lib64/libjulia.so.1 (unknown line)
toarrowtable at /home/ec2-user/.julia/packages/Arrow/CyJ4L/src/write.jl:201
unknown function (ip: 0x7f157f2e5d76)
jl_apply_generic at /usr/bin/../lib64/libjulia.so.1 (unknown line)
macro expansion at /home/ec2-user/.julia/packages/Arrow/CyJ4L/src/write.jl:127 [inlined]
#109 at ./threadingconstructs.jl:169
unknown function (ip: 0x7f157f2fa13c)
jl_apply_generic at /usr/bin/../lib64/libjulia.so.1 (unknown line)
unknown function (ip: 0x7f15bb6b95dd)
unknown function (ip: (nil))
Allocations: 48301139 (Pool: 48286788; Big: 14351); GC: 51
__libc_message at /usr/bin/../lib64/libc.so.6 (unknown line)
malloc_printerr at /usr/bin/../lib64/libc.so.6 (unknown line)
Segmentation fault
@altre
Copy link
Author

altre commented Dec 22, 2020

It also fails with compress=:zstd

@altre altre changed the title Segmentation Fault with Threads.@spawn + Tables.partitioner + write with lz4 compression Segmentation Fault with Threads.@spawn + Tables.partitioner + write with compression Dec 22, 2020
@kobusherbst
Copy link

kobusherbst commented Jan 27, 2021 •
edited
Loading

I have the same problem, this works without compression, but fails with. If one allows only a single thread then it doesn't fail, so perhaps compression is not thread safe?

using Random
using DataFrames
using Arrow
using Tables

function nextidrange(minId, maxId, batchsize, i)
    fromId = minId + batchsize * (i-1)
    toId = min(maxId, (minId + batchsize * i)-1)
    return fromId, toId
end

minId = 1
maxId = 1000
idrange = (maxId - minId) + 1
df = DataFrame(ID=minId:maxId, B=rand(idrange), C=randstring.(fill(5,idrange)));
batchsize = 100
numbatches = ceil(Int32, idrange / batchsize)
partitions = Array{SubDataFrame}(undef, 0)
for i = 1:numbatches 
    fromId, toId = nextidrange(minId, maxId, batchsize, i)
    push!(partitions, filter([:ID] => x -> fromId <= x <= toId, df; view = true))
end
io = IOBuffer()
Arrow.write(io, Tables.partitioner(partitions), compress=:zstd)
seekstart(io)
recordbatches = Arrow.Stream(io)
ab = Array{DataFrame}(undef,0)
for b in recordbatches 
  bt = b |> DataFrame
  println("Rows = $(nrow(bt))")
  push!(ab,bt)
end

@quinnj
Copy link
Member

quinnj commented Jan 29, 2021

Thanks for the reports @kobusherbst and @altre; the compression machinery is indeed not threadsafe, which I've mostly resolve in my local branch, but there's also #108 which is interacting with my testing, so I'm trying to solve both issues in one go here to get threaded writing working reliably. Sorry for the slowness, but I think I'm getting close.

@kobusherbst
Copy link

Thank you @quinnj, both issues are a deal breaker for me in having to deal with huge 600 million plus row datasets.

quinnj added a commit that referenced this issue Jan 30, 2021
@quinnj
Make compressed writing threadsafe
3455304 
Fixes #82. The problem when trying to write arrow using multiple threads and compression was there was only a single compressor object that each thread was simultaneously trying to use. This PR ensures there is a compressor object per thread that will be used per thread.
@quinnj quinnj mentioned this issue Jan 30, 2021
Merged
quinnj added a commit that referenced this issue Jan 30, 2021
@quinnj
Make compressed writing threadsafe (#118)
da73d82 
Fixes #82. The problem when trying to write arrow using multiple threads and compression was there was only a single compressor object that each thread was simultaneously trying to use. This PR ensures there is a compressor object per thread that will be used per thread.
tanmaykm pushed a commit to tanmaykm/arrow-julia that referenced this issue Apr 7, 2021
@quinnj @tanmaykm
Make compressed writing threadsafe (apache#118)
0063a85 
Fixes apache#82. The problem when trying to write arrow using multiple threads and compression was there was only a single compressor object that each thread was simultaneously trying to use. This PR ensures there is a compressor object per thread that will be used per thread.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Make compressed writing threadsafe apache/arrow-julia
3 participants
@quinnj @altre @kobusherbst