[SPARK-46623][CORE][MLLIB][SQL] Replace SimpleDateFormat with DateTimeFormatter #44616
Conversation
|
cc @MaxGekk BTW, we would have to replace |
beliefer
force-pushed
the
replace-sdf-with-dtf
branch
from
c27cdbe to
9f1c744
Compare
beliefer
changed the title
beliefer
force-pushed
the
replace-sdf-with-dtf
branch
from
9f1c744 to
348b333
Compare
beliefer
changed the title
|
cc @mridulm |
There was a problem hiding this comment.
nit: Does not need to be lazy and can be moved to object Master, same in other places as well.
There was a problem hiding this comment.
We dont need this field
| @transient | |
| private val dateTimeFormatter = | |
| DateTimeFormatter | |
| .ofPattern("yyyyMMddHHmmss", Locale.US) | |
| .withZone(ZoneId.systemDefault()) | |
| private val jobTrackerId: String = dateTimeFormatter.format(new Date().toInstant) | |
| private val jobTrackerId: String = { | |
| val dateTimeFormatter = | |
| DateTimeFormatter | |
| .ofPattern("yyyyMMddHHmmss", Locale.US) | |
| .withZone(ZoneId.systemDefault()) | |
| dateTimeFormatter.format(new Date().toInstant) | |
| } |
beliefer
force-pushed
the
replace-sdf-with-dtf
branch
from
348b333 to
f771604
Compare
mridulm
left a comment
mridulm
left a comment
There was a problem hiding this comment.
Looks good to me. +CC @LuciferYang as well.
|
|
||
| private def generateWorkerId(): String = { | ||
| workerIdPattern.format(createDateFormat.format(new Date), host, port) | ||
| workerIdPattern.format(Worker.DATE_TIME_FORMATTER.format(new Date().toInstant), host, port) |
There was a problem hiding this comment.
Is it possible to directly use Instant.now() instead of new Date().toInstant?
| DateTimeFormatter | ||
| .ofPattern("yyyyMMddHHmmss", Locale.US) | ||
| .withZone(ZoneId.systemDefault()) | ||
| dateTimeFormatter.format(new Date().toInstant) |
| val app = new Application("Apache Spark MLlib").setVersion(version) | ||
| val timestamp = new Timestamp() | ||
| .addContent(new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss", Locale.US).format(new Date())) | ||
| .addContent(DATE_TIME_FORMATTER.format(new Date().toInstant)) |
| Timestamp.valueOf( | ||
| TextSocketReader.DATE_FORMAT.format(Calendar.getInstance().getTime())) | ||
| Timestamp.valueOf(TextSocketReader.DATE_TIME_FORMATTER.format( | ||
| Calendar.getInstance().getTime().toInstant)) |
There was a problem hiding this comment.
Is it possible to directly use Instant.now() instead of new alendar.getInstance().getTime().toInstant?
|
|
||
| lazy val externalTempPath: Path = getExternalTmpPath(path) | ||
|
|
||
| private lazy val dateTimeFormatter = |
There was a problem hiding this comment.
This definition can also be moved to object HiveTempPath
|
The GA failure is unrelated. |
|
Merged to master. |
|
I had to manually resolve the jira @beliefer - can you please check if I assigned it to you ? There were two id's with same name. |
|
@mridulm @LuciferYang @HyukjinKwon Thank you! |
I checked. It's OK. Thank you! |
4 participants
What changes were proposed in this pull request?
This PR propose to replace
SimpleDateFormatwithDateTimeFormatter.Why are the changes needed?
According to the javadoc of
SimpleDateFormat, it recommended to useDateTimeFormattertoo.In addition,
DateTimeFormatterhave better performance thanSimpleDateFormattoo.Note:
SimpleDateFormatandDateTimeFormatterare not completely compatible, for example, the formats supported byparseare not exactly the same.Does this PR introduce any user-facing change?
'No'.
How was this patch tested?
GA and manual test.
Was this patch authored or co-authored using generative AI tooling?
'No'.