tables containing Set values are serializable but corresponding deserialized Arrow.Tables are inaccessible

[jrevels]

Running on 32560d3:

julia> using Arrow

julia> construct_table(args...) = (io = IOBuffer(); Arrow.write(io, args...); seekstart(io); Arrow.Table(io))
construct_table (generic function with 1 method)

julia> t = construct_table((sets = [Set([1,2,3]), Set([1,2,3])],));

julia> collect(t.sets)
ERROR: MethodError: Cannot `convert` an object of type Pair{Int64,Nothing} to an object of type Int64
Closest candidates are:
  convert(::Type{T}, ::T) where T<:Number at number.jl:6
  convert(::Type{T}, ::Number) where T<:Number at number.jl:7
  convert(::Type{T}, ::Ptr) where T<:Integer at pointer.jl:23
  ...
Stacktrace:
 [1] setindex!(::Dict{Int64,Nothing}, ::Nothing, ::Pair{Int64,Nothing}) at ./dict.jl:372
 [2] push!(::Set{Int64}, ::Pair{Int64,Nothing}) at ./set.jl:57
 [3] union!(::Set{Int64}, ::Dict{Int64,Nothing}) at ./abstractset.jl:91
 [4] Set{Int64}(::Dict{Int64,Nothing}) at ./set.jl:10
 [5] getindex at /Users/jarrettrevels/.julia/dev/Arrow/src/arraytypes/struct.jl:44 [inlined]
 [6] copyto_unaliased!(::IndexLinear, ::Array{Set{Int64},1}, ::IndexLinear, ::Arrow.Struct{Set{Int64},Tuple{Arrow.Map{Dict{Int64,Nothing},Int32,Arrow.Struct{Arrow.KeyValue{Int64,Nothing},Tuple{Arrow.Primitive{Int64,Array{Int64,1}},Arrow.Struct{Nothing,Tuple{}}}}}}}) at ./abstractarray.jl:860
 [7] copyto! at ./abstractarray.jl:840 [inlined]
 [8] _collect_indices at ./array.jl:642 [inlined]
 [9] collect(::Arrow.Struct{Set{Int64},Tuple{Arrow.Map{Dict{Int64,Nothing},Int32,Arrow.Struct{Arrow.KeyValue{Int64,Nothing},Tuple{Arrow.Primitive{Int64,Array{Int64,1}},Arrow.Struct{Nothing,Tuple{}}}}}}}) at ./array.jl:626
 [10] top-level scope at REPL[36]:1

[quinnj]

Yes, as you've discovered, the Arrow.jl implementation isn't currently very useful for "serialization" of generic objects, but focused on the limited set of types it was originally designed for. I've looked into this and the DataType cases and they're non-trivial with how the code is currently structured, but it should be much nicer if people want to "hook in" to using Arrow.jl for generic serialization. The biggest thing is making the ArrowTypes module much cleaner in terms of the contracts for the various types: what's required and then making sure hte implementation only relies on those contracts. Then people can overload the necessary methods and everything will just work, similar to the StructTypes.jl system. let me look at the NamedTuple issue and see if it's lower hanging fruit

[jrevels]

Ha, yeah - I was actually surprised that Sets were serializable in the first place 😁 Thanks for checking it out.

I'd consider it a win if the error could be moved to serialization time (and try to prevent non-deserializable content from being serialized in the first place as a general rule)

[quinnj]

I'm going to add support for this in #156