Skip to content

[SPARK-35094][SQL]Spark from_json(JsonToStruct) function return wrong value in permissive mode #32252

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Peng-Lei wants to merge 1 commit into from
Closed

[SPARK-35094][SQL]Spark from_json(JsonToStruct) function return wrong value in permissive mode #32252

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -398,8 +398,10 @@ class JacksonParser(
skipRow = structFilters.skipRow(row, index)
} catch {
case e: SparkUpgradeException => throw e
case NonFatal(e) if isRoot =>
badRecordException = badRecordException.orElse(Some(e))
case NonFatal(e) =>
if (isRoot) {
badRecordException = badRecordException.orElse(Some(e))
}
parser.skipChildren()
}
case None =>
Expand Down
30 changes: 28 additions & 2 deletions sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -741,9 +741,10 @@ class JsonFunctionsSuite extends QueryTest with SharedSparkSession {
val df1 = Seq("""{"c2": [19], "c1": 123456}""").toDF("c0")
checkAnswer(df1.select(from_json($"c0", st)), Row(Row(123456, null)))
val df2 = Seq("""{"data": {"c2": [19], "c1": 123456}}""").toDF("c0")
checkAnswer(df2.select(from_json($"c0", new StructType().add("data", st))), Row(Row(null)))
checkAnswer(df2.select(from_json($"c0", new StructType().add("data", st))),
Row(Row(Row(123456, null))))
val df3 = Seq("""[{"c2": [19], "c1": 123456}]""").toDF("c0")
checkAnswer(df3.select(from_json($"c0", ArrayType(st))), Row(null))
checkAnswer(df3.select(from_json($"c0", ArrayType(st))), Row(Seq(Row(123456, null))))
val df4 = Seq("""{"c2": [19]}""").toDF("c0")
checkAnswer(df4.select(from_json($"c0", MapType(StringType, st))), Row(null))
}
Expand Down Expand Up @@ -840,4 +841,29 @@ class JsonFunctionsSuite extends QueryTest with SharedSparkSession {
}
}
}

test("SPARK-35094: Spark from_json(JsonToStruct) function return wrong value " +
Copy link
Member

@HyukjinKwon HyukjinKwon Apr 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Peng-Lei this test passes without your fix. Can you show the reproducible codes with before/after results?

Copy link
Contributor Author

@Peng-Lei Peng-Lei Apr 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@HyukjinKwon I am so sorry, It did pass without my fix. My fix is simply skipping the parsing when an parsing exception occurs.

"in permissive mode in case best effort") {
val s1 = StructField("name", StringType, nullable = true)
val s2_1 =
StructField(
"badNestedField",
StructType(
Seq(StructField("SomethingWhichNotInJsonMessage", IntegerType, nullable = true))))
val s2 =
StructField("nestedField", StructType(Seq(s2_1, s1)))
val customSchema = StructType(Seq(s1, s2))

val jsonStringToTest =
"""{"name":"v1","nestedField":{"badNestedField":"14","name":"v2"}}"""
val df = List(jsonStringToTest)
.toDF("json")
.select(from_json($"json", customSchema).as("toBeFlatten"))
.select("toBeFlatten.*")

assert(
df.select("name").as[String].first() == "v1",
"wrong value in root schema, parser take value from column with same name " +
"but in another nested elvel")
}
}