Skip to content
This repository was archived by the owner on Jun 15, 2021. It is now read-only.

Fix date and timestamp conversion#16

Merged
cmathiesen merged 1 commit intoExpediaGroup:hive-filter-pushdownfrom
rdblue:pr-1326-hive-ppd
Aug 31, 2020
Merged

Fix date and timestamp conversion#16
cmathiesen merged 1 commit intoExpediaGroup:hive-filter-pushdownfrom
rdblue:pr-1326-hive-ppd

Conversation

@rdblue
Copy link

@rdblue rdblue commented Aug 30, 2020

This fixes problems with Date and Timestamp conversion:

  • Dates must be converted to days from epoch, not micros
  • Timestamp#toLocalDate converts with respect to the timestamp's time zone instead of UTC so the date is inaccurate
  • Timestamp access was corrupting the timestamp value and discarding microseconds (HIVE-19726) so this accesses the value directly

This converts directly and fixes the tests using Iceberg literals. The timestamp test was updated to create a Timestamp to pass into Hive directly from a known value in microseconds because parsing is dependent on the local zone. That dependency is okay when Hive is running, but we need a Timestamp with a known UTC instant for the test.

@cmathiesen
Copy link

Thank you very much for helping with all the Timestamp/Date stuff, it all looks good! I think theres just one checkstyle issue with an unused import on the test class which I'll push a change for after merging this :)

@cmathiesen cmathiesen merged commit 614781e into ExpediaGroup:hive-filter-pushdown Aug 31, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments