Support reading decimal arrays from json #3805
Conversation
| @@ -1019,6 +1019,72 @@ impl Decoder { | |||
| )) | |||
| } | |||
|
|
|||
| fn build_decimal128_array( | |||
There was a problem hiding this comment.
FYI Decoder is deprecated and slated to be removed shortly #3610
There was a problem hiding this comment.
I have noticed it. The json reader is used in https://github.com/delta-io/delta-rs/tree/main/rust with the serde Value arguments. Maybe I can find some time to integrate the new interface, but for now it would already help if decimals were supported in the old decoder.
Thank you for your time to review it and point me to the locations of adding the parsing logic. I mostly moved it out of arrow-csv.
arrow-json/src/reader.rs
Outdated
| @@ -3352,4 +3417,72 @@ mod tests { | |||
| let cloned = options.clone(); | |||
| assert_eq!(options, cloned); | |||
| } | |||
|
|
|||
| macro_rules! decimal_json_tests { | |||
There was a problem hiding this comment.
Does this need to be a macro, or could we use a generic function here? They're easier to maintain and debug
There was a problem hiding this comment.
I replaced the macro by a generic function. Maybe the bounds can be written in a better way though.
There was a problem hiding this comment.
I pushed f7e4aa6 which cleans up the constraints a little
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
|
Benchmark runs are scheduled for baseline = 7f460af and contender = 3df7c00. 3df7c00 is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
|
Late, but should Scientific notation also be supported? |
|
I think that would be a valuable further enhancement |
Which issue does this PR close?
Closes #2900
Rationale for this change
The json reader does not support parsing into a decimal array.
What changes are included in this PR?
Support parsing json into decimal array fields.
Are there any user-facing changes?
82408a9?diff=unified#diff-b5489a31aca7aecfa14106360899e04266d43dfdd365638382b0f5bcfb91266bR711
When "123.456" (string) is parsed into a decimal with precision 5, then the 6 at the end will get cut off. I took this behavior from the csv logic and am wondering if it might be worth to round here?