[SPARK-30112][SQL] Allow insert overwrite same table if using dynamic partition overwrite #26752
Conversation
|
cc @cloud-fan |
|
Test build #114806 has finished for PR 26752 at commit
|
viirya
force-pushed
the
dynamic-overwrite-same-table
branch
from
e5f4122 to
378fbc4
Compare
|
Test build #114827 has finished for PR 26752 at commit
|
|
retest this please. |
|
Test build #114840 has finished for PR 26752 at commit
|
|
Retest this please. |
|
Thank you for making a PR for this issue. |
|
|
||
| test("it is allowed to write to a table while querying it for dynamic partition overwrite.") { | ||
| Seq(PartitionOverwriteMode.DYNAMIC.toString, | ||
| PartitionOverwriteMode.STATIC.toString).foreach { usedMode => |
| """ | ||
| |CREATE TABLE insertTable(i int, part1 int, part2 int) USING PARQUET | ||
| |PARTITIONED BY (part1, part2) | ||
| """.stripMargin) |
| """ | ||
| |INSERT OVERWRITE TABLE insertTable PARTITION(part1=1, part2) | ||
| |SELECT i + 1, part2 FROM insertTable | ||
| """.stripMargin) |
|
Test build #114920 has finished for PR 26752 at commit
|
|
Test build #114921 has finished for PR 26752 at commit
|
|
cc @cloud-fan |
|
|
||
| private lazy val parameters = CaseInsensitiveMap(options) | ||
|
|
||
| private[sql] lazy val dynamicPartitionOverwriteEnabled: Boolean = { |
There was a problem hiding this comment.
shall we just name it dynamicPartitionOverwrite? then we can remove https://github.com/apache/spark/pull/26752/files#diff-f9858c2d9d1a3c3e48753ef675bc865aR109
| "INSERT OVERWRITE to a table while querying it should not be allowed.") | ||
| } | ||
|
|
||
| test("it is allowed to write to a table while querying it for dynamic partition overwrite.") { |
There was a problem hiding this comment.
shall we put the JIRA id?
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM. Merged to master.
Thank you so much, @viirya and @cloud-fan .
I also verified the last commit manually.
|
Test build #114956 has finished for PR 26752 at commit
|
|
thanks! @cloud-fan @dongjoon-hyun |
4 participants
What changes were proposed in this pull request?
This patch proposes to allow insert overwrite same table if using dynamic partition overwrite.
Why are the changes needed?
Currently, Insert overwrite cannot overwrite to same table even it is dynamic partition overwrite. But for dynamic partition overwrite, we do not delete partition directories ahead. We write to staging directories and move data to final partition directories. We should be able to insert overwrite to same table under dynamic partition overwrite.
This enables users to read data from a table and insert overwrite to same table by using dynamic partition overwrite. Because this is not allowed for now, users need to write to other temporary location and move it back to the table.
Does this PR introduce any user-facing change?
Yes. Users can insert overwrite same table if using dynamic partition overwrite.
How was this patch tested?
Unit test.