[SPARK-35094][SQL]Spark from_json(JsonToStruct) function return wrong value in permissive mode #32252
Conversation
|
pls change UT to |
Peng-Lei
changed the title
|
@MaxGekk Hi, could you please help to review on this ? thanks |
@AngersZhuuuu done |
|
Can one of the admins verify this patch? |
|
@HyukjinKwon Could you help to review this PR, please. It is related to #30031, I guess. |
| } | ||
| } | ||
|
|
||
| test("SPARK-35094: Spark from_json(JsonToStruct) function return wrong value " + |
There was a problem hiding this comment.
@Peng-Lei this test passes without your fix. Can you show the reproducible codes with before/after results?
There was a problem hiding this comment.
@HyukjinKwon I am so sorry, It did pass without my fix. My fix is simply skipping the parsing when an parsing exception occurs.
|
Hi, All and @HyukjinKwon . |
|
yeah, I think so. |
|
updated the JIRA 👍 |
|
Thank you so much for the confirmation, @HyukjinKwon ! |
6 participants
What changes were proposed in this pull request?
In https://issues.apache.org/jira/browse/SPARK-35094, When use spark from_json(JsonToStruct) function in permissive mode to handle the case of contains incorrect nested json fields. It will return wrong value.
Why are the changes needed?
It is necessary to ensure that the data is correct.
Does this PR introduce any user-facing change?
No
How was this patch tested?
unit test