Skip to content

Conversation

@MaxGekk
Copy link
Member

@MaxGekk MaxGekk commented Dec 12, 2020

What changes were proposed in this pull request?

  1. Move the ALTER TABLE .. DROP PARTITION parsing tests to AlterTableDropPartitionParserSuite
  2. Place v1 tests for ALTER TABLE .. DROP PARTITION from DDLSuite and v2 tests from AlterTablePartitionV2SQLSuite to the common trait AlterTableDropPartitionSuiteBase, so, the tests will run for V1, Hive V1 and V2 DS.

Why are the changes needed?

  • The unification will allow to run common ALTER TABLE .. DROP PARTITION tests for both DSv1 and Hive DSv1, DSv2
  • We can detect missing features and differences between DSv1 and DSv2 implementations.

Does this PR introduce any user-facing change?

No

How was this patch tested?

By running new test suites:

$ build/sbt -Phive -Phive-thriftserver "test:testOnly *AlterTableDropPartitionParserSuite"
$ build/sbt -Phive -Phive-thriftserver "test:testOnly *AlterTableDropPartitionSuite"

@github-actions github-actions bot added the SQL label Dec 12, 2020
sql(s"CREATE TABLE $t (id bigint, data string) $defaultUsing PARTITIONED BY (id)")
sql(s"ALTER TABLE $t ADD PARTITION (id=1) LOCATION 'loc'")

val errMsg = intercept[AnalysisException] {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

V1 and V2 in-memory catalogs throw NoSuchPartitionsException

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is the PR for that: #30778

@SparkQA
Copy link

SparkQA commented Dec 12, 2020

Test build #132699 has finished for PR 30747 at commit 90d167d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

super.test(s"ALTER TABLE .. ADD PARTITION $version: " + testName, testTags: _*)(testFun)
}

protected def withNsTable(ns: String, tableName: String, cat: String = catalog)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will refactor and move this to a common trait later. I don't want to touch tests that are not related to ALTER TABLE .. DROP PARTITION.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea, like adding a DDLCommandTestUtils trait.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cloud-fan Here is the PR for that: #30779

@MaxGekk
Copy link
Member Author

MaxGekk commented Dec 13, 2020

@cloud-fan @HyukjinKwon Please, review this PR.


override def test(testName: String, testTags: Tag*)(testFun: => Any)
(implicit pos: Position): Unit = {
super.test(s"ALTER TABLE .. ADD PARTITION $version: " + testName, testTags: _*)(testFun)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DROP PARTITION

Copy link
Contributor

@cloud-fan cloud-fan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please fix conflicts

…-table-drop-partition-tests

# Conflicts:
#	sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
@SparkQA
Copy link

SparkQA commented Dec 14, 2020

Test build #132773 has finished for PR 30747 at commit 9126bc0.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 14, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37375/

@SparkQA
Copy link

SparkQA commented Dec 14, 2020

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37375/

@SparkQA
Copy link

SparkQA commented Dec 14, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37377/

@SparkQA
Copy link

SparkQA commented Dec 14, 2020

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37377/

@SparkQA
Copy link

SparkQA commented Dec 14, 2020

Test build #132775 has finished for PR 30747 at commit 62f539b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

thanks, merging to master!

@cloud-fan cloud-fan closed this in 141e26d Dec 15, 2020
@MaxGekk MaxGekk deleted the unify-alter-table-drop-partition-tests branch February 19, 2021 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants