[SPARK-33670][SQL] Verify the partition provider is Hive in v1 SHOW TABLE EXTENDED #30618
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Test build #132271 has finished for PR 30618 at commit
|
HyukjinKwon
reviewed
Dec 6, 2020
|
|
||
| class ShowTablesSuite extends ShowTablesSuiteBase with SharedSparkSession | ||
| class ShowTablesSuite extends ShowTablesSuiteBase with SharedSparkSession { | ||
| test("show partitions from a datasource table") { |
Member
Author
There was a problem hiding this comment.
I didn't add because tests from the suite have a prefix already. See the base trait where we override the test() method. Maybe, 2 prefixes are ok.
HyukjinKwon
approved these changes
Dec 6, 2020
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Test build #132284 has finished for PR 30618 at commit
|
Member
|
Merged to master, branch-3.1. |
HyukjinKwon
pushed a commit
that referenced
this pull request
Dec 7, 2020
…ABLE EXTENDED ### What changes were proposed in this pull request? Invoke the check `DDLUtils.verifyPartitionProviderIsHive()` from V1 implementation of `SHOW TABLE EXTENDED` when partition specs are specified. This PR is some kind of follow up #16373 and #15515. ### Why are the changes needed? To output an user friendly error with recommendation like **" ... partition metadata is not stored in the Hive metastore. To import this information into the metastore, run `msck repair table tableName` "** instead of silently output an empty result. ### Does this PR introduce _any_ user-facing change? Yes. ### How was this patch tested? By running the affected test suites, in particular: ``` $ build/sbt -Phive-2.3 -Phive-thriftserver "hive/test:testOnly *PartitionProviderCompatibilitySuite" ``` Closes #30618 from MaxGekk/show-table-extended-verifyPartitionProviderIsHive. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit 29096a8) Signed-off-by: HyukjinKwon <gurwls223@apache.org>
Member
|
@MaxGekk it has a conflict with branch-3.0 and I didn't bother to port back since it's a error message change only. But feel free to open a PR to backport if you feel it's worth. |
MaxGekk
added a commit
to MaxGekk/spark
that referenced
this pull request
Dec 7, 2020
…ABLE EXTENDED Invoke the check `DDLUtils.verifyPartitionProviderIsHive()` from V1 implementation of `SHOW TABLE EXTENDED` when partition specs are specified. This PR is some kind of follow up apache#16373 and apache#15515. To output an user friendly error with recommendation like **" ... partition metadata is not stored in the Hive metastore. To import this information into the metastore, run `msck repair table tableName` "** instead of silently output an empty result. Yes. By running the affected test suites, in particular: ``` $ build/sbt -Phive-2.3 -Phive-thriftserver "hive/test:testOnly *PartitionProviderCompatibilitySuite" ``` Closes apache#30618 from MaxGekk/show-table-extended-verifyPartitionProviderIsHive. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit 29096a8) Signed-off-by: Max Gekk <max.gekk@gmail.com>
MaxGekk
added a commit
to MaxGekk/spark
that referenced
this pull request
Dec 7, 2020
…ABLE EXTENDED Invoke the check `DDLUtils.verifyPartitionProviderIsHive()` from V1 implementation of `SHOW TABLE EXTENDED` when partition specs are specified. This PR is some kind of follow up apache#16373 and apache#15515. To output an user friendly error with recommendation like **" ... partition metadata is not stored in the Hive metastore. To import this information into the metastore, run `msck repair table tableName` "** instead of silently output an empty result. Yes. By running the affected test suites, in particular: ``` $ build/sbt -Phive-2.3 -Phive-thriftserver "hive/test:testOnly *PartitionProviderCompatibilitySuite" ``` Closes apache#30618 from MaxGekk/show-table-extended-verifyPartitionProviderIsHive. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit 29096a8) Signed-off-by: Max Gekk <max.gekk@gmail.com>
Member
Author
|
Here are backports to: |
MaxGekk
deleted the
show-table-extended-verifyPartitionProviderIsHive
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Invoke the check
DDLUtils.verifyPartitionProviderIsHive()from V1 implementation ofSHOW TABLE EXTENDEDwhen partition specs are specified.This PR is some kind of follow up #16373 and #15515.
Why are the changes needed?
To output an user friendly error with recommendation like
"
... partition metadata is not stored in the Hive metastore. To import this information into the metastore, run
msck repair table tableName"
instead of silently output an empty result.
Does this PR introduce any user-facing change?
Yes.
How was this patch tested?
By running the affected test suites, in particular: