Skip to content

Commit 0477d23

Browse files
MaxGekkHyukjinKwon
authored andcommitted
[SPARK-32594][SQL] Fix serialization of dates inserted to Hive tables
### What changes were proposed in this pull request? Fix `DaysWritable` by overriding parent's method `def get(doesTimeMatter: Boolean): Date` from `DateWritable` instead of `Date get()` because the former one uses the first one. The bug occurs because `HiveOutputWriter.write()` call `def get(doesTimeMatter: Boolean): Date` transitively with default implementation from the parent class `DateWritable` which doesn't respect date rebases and uses not initialized `daysSinceEpoch` (0 which `1970-01-01`). ### Why are the changes needed? The changes fix the bug: ```sql spark-sql> CREATE TABLE table1 (d date); spark-sql> INSERT INTO table1 VALUES (date '2020-08-11'); spark-sql> SELECT * FROM table1; 1970-01-01 ``` The expected result of the last SQL statement must be **2020-08-11** but got **1970-01-01**. ### Does this PR introduce _any_ user-facing change? Yes. After the fix, `INSERT` work correctly: ```sql spark-sql> SELECT * FROM table1; 2020-08-11 ``` ### How was this patch tested? Add new test to `HiveSerDeReadWriteSuite` Closes #29409 from MaxGekk/insert-date-into-hive-table. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
1 parent 5d130f0 commit 0477d23

File tree

2 files changed

+11
-1
lines changed

2 files changed

+11
-1
lines changed

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,9 @@ class DaysWritable(
5454
}
5555

5656
override def getDays: Int = julianDays
57-
override def get(): Date = new Date(DateWritable.daysToMillis(julianDays))
57+
override def get(doesTimeMatter: Boolean): Date = {
58+
new Date(DateWritable.daysToMillis(julianDays, doesTimeMatter))
59+
}
5860

5961
override def set(d: Int): Unit = {
6062
gregorianDays = d

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSerDeReadWriteSuite.scala

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -184,4 +184,12 @@ class HiveSerDeReadWriteSuite extends QueryTest with SQLTestUtils with TestHiveS
184184
checkComplexTypes(fileFormat)
185185
}
186186
}
187+
188+
test("SPARK-32594: insert dates to a Hive table") {
189+
withTable("table1") {
190+
sql("CREATE TABLE table1 (d date)")
191+
sql("INSERT INTO table1 VALUES (date '2020-08-11')")
192+
checkAnswer(spark.table("table1"), Row(Date.valueOf("2020-08-11")))
193+
}
194+
}
187195
}

0 commit comments

Comments
 (0)