-
Notifications
You must be signed in to change notification settings - Fork 838
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consolidate JSON Reader options and DecoderOptions
#1539
Conversation
6e5ea8f
to
23c5e4d
Compare
Self { | ||
schema: None, | ||
max_records: None, | ||
batch_size: 1024, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this default is moved into DecoderOptions
23c5e4d
to
ac9c647
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It took me a while to work out why there is a separation between the Decoder
and a Reader
and therefore we end up with nested builders, but I think this makes sense. We want to allow people to construct a Decoder
with arbitrary streams of serde_json::Value
and convert them to RecordBatch
, whereas Reader
only supports files 👍
I tried to encode this insight into the docs here: #1559 |
DecoderOptions
Which issue does this PR close?
Draft as it builds on #1451 from @sum12 .Closes #1538
Rationale for this change
@sum12 started refactoring options into a common struct in #1451 and I wanted to finish that work to minimize API churn when we release the next arrow
What changes are included in this PR?
batch_size
to DecoderOptions as well.ReaderBuilder
Are there any user-facing changes?
Yes -- if they use the low level Json decoder, the options are now encoded in a struct rather than as parameters