add Arrow.jl conversion methods so that Samples/SamplesInfo can be (de)serialized as Arrow values #68
Conversation
jrevels
force-pushed
the
jr/arrowconvert
branch
from
df4e1a0 to
11e0f10
Compare
|
Looks like upgrading to Arrow 1.3 surfaced a DataFrames bug fixed upstream by JuliaData/DataFrames.jl#2682? At least, I think that's what's going on...locally using DataFrames' |
jrevels
force-pushed
the
jr/arrowconvert
branch
from
0944386 to
306cc3f
Compare
…e)serialized as Arrow values
jrevels
force-pushed
the
jr/arrowconvert
branch
from
7e7ab51 to
6b0afc0
Compare
| end | ||
|
|
||
| function Arrow.ArrowTypes.fromarrow(::Type{<:Samples}, arrow_data, arrow_info, arrow_encoded) | ||
| info = SamplesInfo(arrow_info; validate=false) |
There was a problem hiding this comment.
I wonder if validate=false is the right choice? I guess we can assume it's properly formulated since usually we will be validating on construction and then serializing/deserializing
There was a problem hiding this comment.
I guess we can assume it's properly formulated since usually we will be validating on construction and then serializing/deserializing
Yeah, this is the assumption we're making
Yup :) Though this is more useful to enable conveniently (de)serialization of individual sample data segments to/from Arrow (either for storage or IPC) If you wanted to use Arrow as a storage format for whole sample data files w/ Onda, it'd make more sense to add an |
requires apache/arrow-julia#150this now requires (and is being periodically re-semi-updated against) apache/arrow-julia#156, which supersedes apache/arrow-julia#150