Skip to content

Commit 78f7c30

Browse files
nikolamand-dbMaxGekk
authored andcommitted
[SPARK-42328][SQL] Remove _LEGACY_ERROR_TEMP_1175 from error classes
### What changes were proposed in this pull request? Only occurrence of `_LEGACY_ERROR_TEMP_1175` appears under conversion from Spark data types to Parquet. All supported documented [Spark data types](https://spark.apache.org/docs/latest/sql-ref-datatypes.html) are covered in the [conversion function](https://github.com/apache/spark/blob/3e0808c33f185c13808ce2d547ce9ba0057d31a6/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L517-L745) (`VarcharType` and `CharType` are not present but they are passed down as string before reaching the conversion function), so under normal circumstances user can't force this error. Convert the error class to `INTERNAL_ERROR`. ### Why are the changes needed? Remove legacy error classes as part of activity in [SPARK-37935](https://issues.apache.org/jira/browse/SPARK-37935). ### Does this PR introduce _any_ user-facing change? If the Spark works correctly, user shouldn't be able to run into `INTERNAL_ERROR` by using the public API. ### How was this patch tested? Added test to `QueryCompilationErrorsSuite` and tested with sbt: ``` project sql testOnly *QueryCompilationErrorsSuite ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #45183 from nikolamand-db/nikolamand-db/SPARK-42328. Authored-by: Nikola Mandic <nikola.mandic@databricks.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>
1 parent df4d489 commit 78f7c30

File tree

3 files changed

+22
-7
lines changed

3 files changed

+22
-7
lines changed

common/utils/src/main/resources/error/error-classes.json

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -5118,11 +5118,6 @@
51185118
"Unrecognized Parquet type: <field>."
51195119
]
51205120
},
5121-
"_LEGACY_ERROR_TEMP_1175" : {
5122-
"message" : [
5123-
"Unsupported data type <dataType>."
5124-
]
5125-
},
51265121
"_LEGACY_ERROR_TEMP_1181" : {
51275122
"message" : [
51285123
"Stream-stream join without equality predicate is not supported."

sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1908,8 +1908,9 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase with Compilat
19081908

19091909
def cannotConvertDataTypeToParquetTypeError(field: StructField): Throwable = {
19101910
new AnalysisException(
1911-
errorClass = "_LEGACY_ERROR_TEMP_1175",
1912-
messageParameters = Map("dataType" -> field.dataType.catalogString))
1911+
errorClass = "INTERNAL_ERROR",
1912+
messageParameters = Map("message" ->
1913+
s"Cannot convert Spark data type ${toSQLType(field.dataType)} to any Parquet type."))
19131914
}
19141915

19151916
def incompatibleViewSchemaChangeError(

sql/core/src/test/scala/org/apache/spark/sql/errors/QueryCompilationErrorsSuite.scala

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ import org.apache.spark.sql._
2424
import org.apache.spark.sql.api.java.{UDF1, UDF2, UDF23Test}
2525
import org.apache.spark.sql.catalyst.expressions.{Coalesce, Literal, UnsafeRow}
2626
import org.apache.spark.sql.catalyst.parser.ParseException
27+
import org.apache.spark.sql.execution.datasources.parquet.SparkToParquetSchemaConverter
2728
import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
2829
import org.apache.spark.sql.expressions.SparkUserDefinedFunction
2930
import org.apache.spark.sql.functions._
@@ -962,6 +963,24 @@ class QueryCompilationErrorsSuite
962963
"methodName" -> "update",
963964
"className" -> "org.apache.spark.sql.catalyst.expressions.UnsafeRow"))
964965
}
966+
967+
test("INTERNAL_ERROR: Convert unsupported data type from Spark to Parquet") {
968+
val converter = new SparkToParquetSchemaConverter
969+
val dummyDataType = new DataType {
970+
override def defaultSize: Int = 0
971+
972+
override def simpleString: String = "Dummy"
973+
974+
override private[spark] def asNullable = NullType
975+
}
976+
checkError(
977+
exception = intercept[AnalysisException] {
978+
converter.convertField(StructField("test", dummyDataType))
979+
},
980+
errorClass = "INTERNAL_ERROR",
981+
parameters = Map("message" -> "Cannot convert Spark data type \"DUMMY\" to any Parquet type.")
982+
)
983+
}
965984
}
966985

967986
class MyCastToString extends SparkUserDefinedFunction(

0 commit comments

Comments
 (0)