[SPARK-21365][PYTHON] Deduplicate logics parsing DDL type/schema definition

[HyukjinKwon]

What changes were proposed in this pull request?

This PR deals with four points as below:

• Reuse existing DDL parser APIs rather than reimplementing within PySpark

• Support DDL formatted string, field type, field type.

• Support case-insensitivity for parsing.

• Support nested data types as below:

  Before

  >>> spark.createDataFrame([[[1]]], "struct<a: struct<b: int>>").show()
  ...
  ValueError: The strcut field string format is: 'field_name:field_type', but got: a: struct<b: int>
  

  >>> spark.createDataFrame([[[1]]], "a: struct<b: int>").show()
  ...
  ValueError: The strcut field string format is: 'field_name:field_type', but got: a: struct<b: int>
  

  >>> spark.createDataFrame([[1]], "a int").show()
  ...
  ValueError: Could not parse datatype: a int
  

  After

  >>> spark.createDataFrame([[[1]]], "struct<a: struct<b: int>>").show()
  +---+
  |  a|
  +---+
  |[1]|
  +---+
  

  >>> spark.createDataFrame([[[1]]], "a: struct<b: int>").show()
  +---+
  |  a|
  +---+
  |[1]|
  +---+
  

  >>> spark.createDataFrame([[1]], "a int").show()
  +---+
  |  a|
  +---+
  |  1|
  +---+
  

How was this patch tested?

[HyukjinKwon]

Without this, I should do something like ...

getattr(getattr(sc._jvm.org.apache.spark.sql.catalyst.parser, "CatalystSqlParser$"), "MODULE$").parseDataType("a")

[HyukjinKwon]

So, if I haven't missed anything, this PR drops the support the type parsing null. I guess it is almost seldom that we explicitly set the type with null. Also, IIRC, we will support NullType via void (SPARK-20680) soon as a workaround.

[HyukjinKwon]

cc @cloud-fan, @felixcheung and @zero323 who I remember I talked with about similar issues before.

[felixcheung]

Add
@gatorsmile
@holdenk

[SparkQA]

Test build #79470 has finished for PR 18590 at commit 3472873.

• This patch fails SparkR unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

retest this please

[SparkQA]

Test build #79481 has finished for PR 18590 at commit 3472873.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #79492 has finished for PR 18590 at commit 9d857e6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

won't fieldname: datatype, fieldname: datatype be parsed as DDL schema?

[HyukjinKwon]

I tested few cases but it looks not:

scala> StructType.fromDDL("a struct<a: INT, b: STRING>")
res5: org.apache.spark.sql.types.StructType = StructType(StructField(a,StructType(StructField(a,IntegerType,true), StructField(b,StringType,true)),true))

scala> StructType.fromDDL("a INT, b STRING")
res6: org.apache.spark.sql.types.StructType = StructType(StructField(a,IntegerType,true), StructField(b,StringType,true))

scala> StructType.fromDDL("a: INT, b: STRING")
org.apache.spark.sql.catalyst.parser.ParseException:
extraneous input ':' expecting ...

[holdenk]

Thanks for working on this, unifiying the parsing logic to be Scala side seems like a good idea.

[holdenk]

We have pretty similar logic bellow, would it make sense to think about if there is a nicer more general way to handle these delayed iniatilization classes?

[HyukjinKwon]

hmm.. I tried several ways I could think at my best but I could not figure out ...

[cloud-fan]

LGTM, merging to master!