Skip to content

Commit ac10d8e

Browse files
committed
add migration guide
1 parent b33066f commit ac10d8e

File tree

1 file changed

+4
-0
lines changed

1 file changed

+4
-0
lines changed

docs/sql-migration-guide.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,10 @@ license: |
3636

3737
- In Spark 3.1, NULL elements of structures, arrays and maps are converted to "null" in casting them to strings. In Spark 3.0 or earlier, NULL elements are converted to empty strings. To restore the behavior before Spark 3.1, you can set `spark.sql.legacy.castComplexTypesToString.enabled` to `true`.
3838

39+
- In Spark 3.1, when `spark.sql.ansi.enabled` is false, sum aggregation of decimal type column always returns `null` on decimal value overflow. In Spark 3.0 or earlier, when `spark.sql.ansi.enabled` is false and decimal value overflow happens in sum aggregation of decimal type column:
40+
- If it is hash aggregation with `group by` clause, a runtime exception is thrown.
41+
- Otherwise, null is returned.
42+
3943
## Upgrading from Spark SQL 3.0 to 3.0.1
4044

4145
- In Spark 3.0, JSON datasource and JSON function `schema_of_json` infer TimestampType from string values if they match to the pattern defined by the JSON option `timestampFormat`. Since version 3.0.1, the timestamp type inference is disabled by default. Set the JSON option `inferTimestamp` to `true` to enable such type inference.

0 commit comments

Comments
 (0)