[SPARK-40819][SQL][3.3] Timestamp nanos behaviour regression #39904
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
HyukjinKwon
approved these changes
Feb 6, 2023
HyukjinKwon
changed the title
dongjoon-hyun
approved these changes
Feb 7, 2023
There was a problem hiding this comment.
+1, LGTM (Pending CIs).
cc @viirya since he is the release manager of Apache Spark 3.3.2.
dongjoon-hyun
changed the title
viirya
reviewed
Feb 7, 2023
Comment on lines
+3462
to
+3467
| val LEGACY_PARQUET_NANOS_AS_LONG = buildConf("spark.sql.legacy.parquet.nanosAsLong") | ||
| .internal() | ||
| .doc("When true, the Parquet's nanos precision timestamps are converted to SQL long values.") | ||
| .version("3.2.3") | ||
| .booleanConf | ||
| .createWithDefault(false) |
There was a problem hiding this comment.
Seems this is worth documenting it in migration guide?
There was a problem hiding this comment.
I will look at adding to the guide
viirya
approved these changes
Feb 7, 2023
|
Merged to branch-3.3. |
HyukjinKwon
pushed a commit
that referenced
this pull request
Feb 8, 2023
As per HyukjinKwon request on #38312 to backport fix into 3.3 ### What changes were proposed in this pull request? Handle `TimeUnit.NANOS` for parquet `Timestamps` addressing a regression in behaviour since 3.2 ### Why are the changes needed? Since version 3.2 reading parquet files that contain attributes with type `TIMESTAMP(NANOS,true)` is not possible as ParquetSchemaConverter returns ``` Caused by: org.apache.spark.sql.AnalysisException: Illegal Parquet type: INT64 (TIMESTAMP(NANOS,true)) ``` https://issues.apache.org/jira/browse/SPARK-34661 introduced a change matching on the `LogicalTypeAnnotation` which only covers Timestamp cases for `TimeUnit.MILLIS` and `TimeUnit.MICROS` meaning `TimeUnit.NANOS` would return `illegalType()` Prior to 3.2 the matching used the `originalType` which for `TIMESTAMP(NANOS,true)` return `null` and therefore resulted to a `LongType`, the change proposed is too consider `TimeUnit.NANOS` and return `LongType` making behaviour the same as before. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Added unit test covering this scenario. Internally deployed to read parquet files that contain `TIMESTAMP(NANOS,true)` Closes #39904 from awdavidson/ts-nanos-fix-3.3. Lead-authored-by: alfreddavidson <alfie.davidson9@gmail.com> Co-authored-by: awdavidson <54780428+awdavidson@users.noreply.github.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As per @HyukjinKwon request on #38312 to backport fix into 3.3
What changes were proposed in this pull request?
Handle
TimeUnit.NANOSfor parquetTimestampsaddressing a regression in behaviour since 3.2Why are the changes needed?
Since version 3.2 reading parquet files that contain attributes with type
TIMESTAMP(NANOS,true)is not possible as ParquetSchemaConverter returnshttps://issues.apache.org/jira/browse/SPARK-34661 introduced a change matching on the
LogicalTypeAnnotationwhich only covers Timestamp cases forTimeUnit.MILLISandTimeUnit.MICROSmeaningTimeUnit.NANOSwould returnillegalType()Prior to 3.2 the matching used the
originalTypewhich forTIMESTAMP(NANOS,true)returnnulland therefore resulted to aLongType, the change proposed is too considerTimeUnit.NANOSand returnLongTypemaking behaviour the same as before.Does this PR introduce any user-facing change?
No
How was this patch tested?
Added unit test covering this scenario.
Internally deployed to read parquet files that contain
TIMESTAMP(NANOS,true)