From 42780a3fc61fad60fbde12dd7589983c90661916 Mon Sep 17 00:00:00 2001 From: PHILO-HE Date: Wed, 12 Jan 2022 17:09:56 +0800 Subject: [PATCH] [NSE-635] Add document to clarify incompatibility issues in expressions (#657) * Initial commit * Clarify incompatibility cases in casting string to int/bigint/float4/float8 * Update the table * Rename the doc * Update the doc --- docs/Expressions-Compatibility.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 docs/Expressions-Compatibility.md diff --git a/docs/Expressions-Compatibility.md b/docs/Expressions-Compatibility.md new file mode 100644 index 000000000..2d8d3b29e --- /dev/null +++ b/docs/Expressions-Compatibility.md @@ -0,0 +1,16 @@ +## Gazelle Expression Compatibility with Apache Spark + +There are some cases that Gazelle behaves differently from Apache Spark. Here, we list the compatibility issues we have not addressed so far. + + +| No. | Expression | Incompatibility | +| --- | ----------------------------------------|---------------------------------------------------------------------| +| 1 | all expressions | Incompatibility issue when ANSI is on (throw exceptions at runtime instead of return null). | +| 2 | get_json_object | Single quote mark is not supported, but only support regular double quote mark.
| +| | | If multiple same keys are contained, null will be returned. But vanilla spark returns the value for the firstly emerged key.| +| 3 | from_unixtime | Specifying timezone is not supported. By default, return date for
UTC, not for local timezone like vanilla spark. | | +| 4 | date/time related expressions | Incompatible behaviors for different LEGACY_TIME_PARSER_POLICY
(corrected, exception, legacy). | +| 5 | expressions with date format provided. | Parsing user-specified date format is not well supported. | +| 6 | castINT/castBIGINT/castFLOAT4/castFLOAT8| Return digital part leading in strings like "123abc" in WSCG, but vanilla spark return null. | + +