Update from_json to use new cudf features #11497
Conversation
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
|
The dependency was merged in yesterday so moving to ready for review |
|
build |
| @@ -336,6 +336,8 @@ object GpuJsonReadCommon { | |||
| .withLeadingZeros(options.allowNumericLeadingZeros) | |||
| .withNonNumericNumbers(options.allowNonNumericNumbers) | |||
| .withUnquotedControlChars(allowUnquotedControlChars) | |||
| .withCudfPruneSchema(true) | |||
| .withExperimental(true) | |||
There was a problem hiding this comment.
curious on what the plan is for the withExperimental flag to be deprecated.
As it is here, I don't know what it does, but the cuDF code does say that it enables features such as utf-8 and new column tree construction. Is that the extent? Should we add a comment here that says why we set it?
There was a problem hiding this comment.
The experimental flag enables most of the fixes. They are things that should be okay to work with the python side of things, but to reduce the risk for the 24.10 release they were put under this flag. It is likely that it will be removed at some point in 24.12, but we can wait and see
|
looks like I missed some test failures in 3.2.0. I will investigate what changed so that we can get the proper things filed. |
|
build |
|
@abellina please take another look |
sameerz
added
the
task
This depends on rapidsai/cudf#16545 but enables a lot more tests to pass. This does not yet enable from_json by default, but a follow on issue probably will with some restrictions on the types that are supported.