[SPARK-40820][PYTHON][SQL] Creating StructType from Json

[anthonywainer]

What changes were proposed in this pull request?
Read schema from json without nullable and metadata

Why are the changes needed?
In order to read schema from json and avoid having to set implicit values

Does this PR introduce any user-facing change?
Yes, avoiding filling json with implicit values

How was this patch tested?
Unit tests

When create a StructType from a Python dictionary you use StructType.fromJson or in scala DataType.fromJson

To create a schema can be created as follows from the code below, but it requires to put inside the json: Nullable and Metadata, this is inconsistent because within the DataType class this by default.

schema = {
            "name": "name",
            "type": "string"
        }

StructField.fromJson(schema)

Python Error:

from pyspark.sql.types import StructField
schema = {
            "name": "c1",
            "type": "string"
        }
StructField.fromJson(schema)

>>
Traceback (most recent call last):
  File "code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "pyspark/sql/types.py", line 583, in fromJson
    json["nullable"],
KeyError: 'nullable' 

Scala Error:

    val schema =
      """
        |{
        |    "type": "struct",
        |    "fields": [
        |        {
        |            "name": "c1",
        |            "type": "string",
        |            "nullable": false
        |        }
        |    ]
        |}
        |""".stripMargin
    DataType.fromJson(schema)

>>
Failed to convert the JSON string '{"name":"c1","type":"string"}' to a field.
java.lang.IllegalArgumentException: Failed to convert the JSON string '{"name":"c1","type":"string"}' to a field.
	at org.apache.spark.sql.types.DataType$.parseStructField(DataType.scala:268)
	at org.apache.spark.sql.types.DataType$.$anonfun$parseDataType$1(DataType.scala:225)

[anthonywainer]

@HyukjinKwon I have re-opened the PR, could you check please?

[HyukjinKwon]

Let's move the tests to DataTypeSuite.scala

[anthonywainer]

moved!

[HyukjinKwon]

Suggested change

	def test_struct_field_from_json(self):
	def test_struct_field_from_json(self):
	# SPARK-40820: fromJson with only name and type

[anthonywainer]

changed!

[anthonywainer]

@HyukjinKwon could you check this, please?

[HyukjinKwon]

Merged to master.