Spark 4: Fix parquet_test.py [databricks] #11519
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes NVIDIA#11015. Contributes to NVIDIA#11004. This commit addresses the tests that fail in parquet_test.py, when run on Spark 4. 1. Some of the tests were failing as a result of NVIDIA#5114. Those tests have been disabled, at least until we get around to supporting aggregations with ANSI mode enabled. 2. `test_parquet_check_schema_compatibility` fails on Spark 4 regardless of ANSI mode, because it tests implicit type promotions where the read schema includes wider columns than the write schema. This will require new code. The test is disabled until NVIDIA#11512 is addressed. 3. `test_parquet_int32_downcast` had an erroneous setup phase that fails in ANSI mode. This has been corrected. The test was refactored to run in ANSI and non-ANSI mode. Signed-off-by: MithunR <mithunr@nvidia.com>
|
Build |
|
Thanks for this. The fix for |
razajafri
previously approved these changes
Oct 2, 2024
|
@razajafri: I've modified this PR to also include a fix for #11531. |
mythrocks
changed the title
|
Build |
I have tested that this addresses the test failure in Databricks 14.3. In making this change, I had to dismiss your earlier review. If you wouldn't mind taking another look, I'd appreciate it. |
razajafri
approved these changes
Oct 7, 2024
|
This has been merged. Thank you for the review, @razajafri. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #11015.
Fixes #11531.
Contributes to #11004 and #10661.
This commit addresses the tests that fail in parquet_test.py, when run on Spark 4.
Some of the tests were failing as a result of [FEA] Support aggregates when ANSI mode is enabled #5114. Those tests have been disabled, at least until we get around to supporting aggregations with ANSI mode enabled.
test_parquet_check_schema_compatibilityfails on Spark 4 regardless of ANSI mode, because it tests implicit type promotions where the read schema includes wider columns than the write schema. This will require new code. The test is disabled until [BUG] Support for wider types in read schemas for Parquet Reads #11512 is addressed.test_parquet_int32_downcasthad an erroneous setup phase that fails in ANSI mode. This has been corrected. The test was refactored to run in ANSI and non-ANSI mode.