Skip to content

[SPARK-26593][SQL] Use Proleptic Gregorian calendar in casting UTF8String to Date/TimestampType #23512

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 21 commits into from
Closed

[SPARK-26593][SQL] Use Proleptic Gregorian calendar in casting UTF8String to Date/TimestampType #23512

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
c5592d8
Initial implementation
MaxGekk Jan 9, 2019
c22b15c
Fix time only parsing
MaxGekk Jan 10, 2019
89be967
Fix Hive hash for old date
MaxGekk Jan 10, 2019
6d9dc96
Fix coding style
MaxGekk Jan 10, 2019
0bc1470
Using Proleptic Gregorian calendar in stringToDate
MaxGekk Jan 10, 2019
2c811af
Reusing methods from DateTimeUtils
MaxGekk Jan 10, 2019
b9a52a8
Using cast instead of casting Timestamp
MaxGekk Jan 11, 2019
c2bc4b0
Merge remote-tracking branch 'origin/master' into utf8string-timestam…
MaxGekk Jan 11, 2019
00ed08c
Porting getYear on Proleptic Gregorian cal
MaxGekk Jan 11, 2019
f794378
Re-implementing splitDate
MaxGekk Jan 12, 2019
0f7ddd2
Avoid using SimpleDateFormat for old dates
MaxGekk Jan 12, 2019
70aa674
Removing unused stringToTime
MaxGekk Jan 13, 2019
8ed3f45
Fix imports
MaxGekk Jan 13, 2019
2ebc272
Improve getLastDayOfMonth
MaxGekk Jan 13, 2019
61ad31a
Using Year.isLeap
MaxGekk Jan 13, 2019
0a89446
Fix getLastDayOfMonth
MaxGekk Jan 13, 2019
336b9ac
Merge remote-tracking branch 'origin/master' into utf8string-timestam…
MaxGekk Jan 14, 2019
f048a3d
Fix merge
MaxGekk Jan 14, 2019
f02c85c
Minor refactoring
MaxGekk Jan 14, 2019
831cdd1
Avoid SimpleDateFormat in the suite
MaxGekk Jan 16, 2019
1af865f
Remove SimpleDateFormat from the suite
MaxGekk Jan 16, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ package org.apache.spark.sql.catalyst.util
import java.time.{Instant, ZoneId}
import java.util.Locale

import org.apache.spark.sql.catalyst.util.DateTimeUtils.instantToDays

sealed trait DateFormatter extends Serializable {
def parse(s: String): Int // returns days since epoch
def format(days: Int): String
Expand All @@ -38,11 +40,7 @@ class Iso8601DateFormatter(
toInstantWithZoneId(temporalAccessor, UTC)
}

override def parse(s: String): Int = {
val seconds = toInstant(s).getEpochSecond
val days = Math.floorDiv(seconds, DateTimeUtils.SECONDS_PER_DAY)
days.toInt
}
override def parse(s: String): Int = instantToDays(toInstant(s))

override def format(days: Int): String = {
val instant = Instant.ofEpochSecond(days * DateTimeUtils.SECONDS_PER_DAY)
Expand Down
Loading