Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] CAST string to timestamp produces null for some valid timestamps #2889

Open
andygrove opened this issue Jul 8, 2021 · 2 comments
Open
Labels
bug Something isn't working

Comments

@andygrove
Copy link
Contributor

Describe the bug
Casting the following strings to timestamp produces null values instead of valid timestamps.

"\nT2\n", // today, 02:00:00.0
"\nT\n", // today, midnight
"\n:\n", // today, midnight
"\n1:\n", // today, 01:00:00.0
"\n5:24\n", // today, 05:24:00.0
"\n:2\n", // today, 2 minutes after midnight
"\n1::2\n" // today, 01:00:02.0

Steps/Code to reproduce bug

I will check in the fuzzer used to generate these values as part of the PR that fixes the issue.

Expected behavior
We should produce the same values as Spark.

Additional context
None

@andygrove andygrove added bug Something isn't working ? - Needs Triage Need team to review and classify labels Jul 8, 2021
@andygrove andygrove added this to the July 5 - July 16 milestone Jul 8, 2021
@andygrove andygrove self-assigned this Jul 8, 2021
@Salonijain27 Salonijain27 removed the ? - Needs Triage Need team to review and classify label Jul 13, 2021
@sameerz
Copy link
Collaborator

sameerz commented Jul 20, 2021

This will be partially addressed in 21.08 and completed in 21.10.

@andygrove
Copy link
Contributor Author

Rather than try and support these edge cases I am now leaning towards having a regex to detect them instead and fail if we see them and tell the user to disable spark.rapids.sql.castStringToTimestamp.enabled if they run into this error.

@andygrove andygrove removed this from the Aug 2 - Aug 13 milestone Aug 13, 2021
@andygrove andygrove added the ? - Needs Triage Need team to review and classify label Aug 13, 2021
@Salonijain27 Salonijain27 removed the ? - Needs Triage Need team to review and classify label Aug 18, 2021
@andygrove andygrove removed their assignment Aug 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants