[SPARK-26389][SS] Add force delete temp checkpoint configuration #23732

gaborgsomogyi · 2019-02-04T16:24:32Z

What changes were proposed in this pull request?

Not all users wants to keep temporary checkpoint directories. Additionally hard to restore from it.

In this PR I've added a force delete flag which is default false. Additionally not clear for users when temporary checkpoint directory deleted so added log messages to explain this a bit more.

How was this patch tested?

Existing + additional unit tests.

SparkQA · 2019-02-04T18:26:56Z

Test build #102036 has finished for PR 23732 at commit 7c304f6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gaborgsomogyi · 2019-02-04T18:39:36Z

retest this please

SparkQA · 2019-02-04T20:35:00Z

Test build #102039 has finished for PR 23732 at commit 7c304f6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gaborgsomogyi · 2019-02-04T20:40:43Z

retest this please

SparkQA · 2019-02-05T00:51:41Z

Test build #102042 has finished for PR 23732 at commit 7c304f6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

camper42 · 2019-02-05T04:11:04Z

thanks for quick improvement

gaborgsomogyi · 2019-02-05T08:45:41Z

cc @zsxwing @jose-torres

jose-torres · 2019-02-06T18:20:26Z

Two minor comments:

The flag description is a bit confusing; I would just describe the behavior when the flag is on, rather than saying it’s “normal” deletion.

I think the new flag should be read in DataStreamWriter, so the shouldDelete value we pass into StreamExecution will always be correct.

gaborgsomogyi · 2019-02-06T18:53:09Z

The flag description is a bit confusing; I would just describe the behavior when the flag is on, rather than saying it’s “normal” deletion.

Clear, will do.

I think the new flag should be read in DataStreamWriter, so the shouldDelete value we pass into StreamExecution will always be correct.

shouldDelete flag is actually shouldDeleteIfQueryNotFailed and DataStreamWriter doesn't know whether the query stopped without exception. Do you mean change actual behaviour?

sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala

sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala

jose-torres · 2019-02-07T18:00:12Z

You're right, there's no good way to move the flag the way I wanted. Looks good modulo Dongjoon's comments.

SparkQA · 2019-02-07T18:54:38Z

Test build #102078 has finished for PR 23732 at commit ca77ebc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-02-08T00:44:41Z

Test build #102083 has finished for PR 23732 at commit ce2577d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR

Looks good overall. Left minor comment on documentation.

HeartSaVioR · 2019-02-08T01:32:02Z

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

@@ -907,6 +907,12 @@ object SQLConf {
    .stringConf
    .createOptional

+  val FORCE_DELETE_TEMP_CHECKPOINT_LOCATION =
+    buildConf("spark.sql.streaming.forceDeleteTempCheckpointLocation")
+      .doc("When true, enable temporary checkpoint locations force delete.")


nit: I'd like to have this to be a valid and more described sentence like other configurations, especially this feature is only documented here. One example for me, When true, Spark always deletes temporary checkpoint locations for success queries.

Ur, @HeartSaVioR . That is the behavior before this PR.
The benefit of this configuration is to delete the locations when exception occurs.

Ah yes my bad I meant failed queries, maybe better, regardless of query's result - success or failure. Anyway this option is only documented here so I'd just wanted to make sure there's clear description.

dongjoon-hyun

+1, LGTM. This configuration is useful.
Merged to master.

dongjoon-hyun · 2019-02-08T18:31:40Z

Thank you, @gaborgsomogyi , @camper42 , @jose-torres , @HeartSaVioR !

## What changes were proposed in this pull request? Not all users wants to keep temporary checkpoint directories. Additionally hard to restore from it. In this PR I've added a force delete flag which is default `false`. Additionally not clear for users when temporary checkpoint directory deleted so added log messages to explain this a bit more. ## How was this patch tested? Existing + additional unit tests. Closes apache#23732 from gaborgsomogyi/SPARK-26389. Authored-by: Gabor Somogyi <gabor.g.somogyi@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

gatorsmile · 2019-12-23T02:33:08Z

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

@@ -907,6 +907,12 @@ object SQLConf {
    .stringConf
    .createOptional

+  val FORCE_DELETE_TEMP_CHECKPOINT_LOCATION =
+    buildConf("spark.sql.streaming.forceDeleteTempCheckpointLocation")


Let us follow the other boolean conf naming convention?

spark.sql.streaming.forceDeleteTempCheckpointLocation.enabled?

cc @Ngone51 could you submit a PR to make a change?

Submitted #26981

[SPARK-26389][SS] Add force delete temp checkpoint configuration

7c304f6

Doc fic

ca77ebc

dongjoon-hyun reviewed Feb 7, 2019

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Feb 7, 2019

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala Outdated Show resolved Hide resolved

Fix

ce2577d

HeartSaVioR reviewed Feb 8, 2019

View reviewed changes

dongjoon-hyun approved these changes Feb 8, 2019

View reviewed changes

dongjoon-hyun closed this in 701b06a Feb 8, 2019

gatorsmile reviewed Dec 23, 2019

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-26389][SS] Add force delete temp checkpoint configuration #23732

[SPARK-26389][SS] Add force delete temp checkpoint configuration #23732

gaborgsomogyi commented Feb 4, 2019

SparkQA commented Feb 4, 2019

gaborgsomogyi commented Feb 4, 2019

SparkQA commented Feb 4, 2019

gaborgsomogyi commented Feb 4, 2019

SparkQA commented Feb 5, 2019

camper42 commented Feb 5, 2019 •

edited

Loading

gaborgsomogyi commented Feb 5, 2019

jose-torres commented Feb 6, 2019

gaborgsomogyi commented Feb 6, 2019 •

edited

Loading

jose-torres commented Feb 7, 2019

SparkQA commented Feb 7, 2019

SparkQA commented Feb 8, 2019

HeartSaVioR left a comment

HeartSaVioR Feb 8, 2019

dongjoon-hyun Feb 8, 2019

HeartSaVioR Feb 8, 2019 •

edited

Loading

dongjoon-hyun left a comment

dongjoon-hyun commented Feb 8, 2019

gatorsmile Dec 23, 2019

gatorsmile Dec 23, 2019

Ngone51 Dec 23, 2019

[SPARK-26389][SS] Add force delete temp checkpoint configuration #23732

[SPARK-26389][SS] Add force delete temp checkpoint configuration #23732

Conversation

gaborgsomogyi commented Feb 4, 2019

What changes were proposed in this pull request?

How was this patch tested?

SparkQA commented Feb 4, 2019

gaborgsomogyi commented Feb 4, 2019

SparkQA commented Feb 4, 2019

gaborgsomogyi commented Feb 4, 2019

SparkQA commented Feb 5, 2019

camper42 commented Feb 5, 2019 • edited Loading

gaborgsomogyi commented Feb 5, 2019

jose-torres commented Feb 6, 2019

gaborgsomogyi commented Feb 6, 2019 • edited Loading

jose-torres commented Feb 7, 2019

SparkQA commented Feb 7, 2019

SparkQA commented Feb 8, 2019

HeartSaVioR left a comment

Choose a reason for hiding this comment

HeartSaVioR Feb 8, 2019

Choose a reason for hiding this comment

dongjoon-hyun Feb 8, 2019

Choose a reason for hiding this comment

HeartSaVioR Feb 8, 2019 • edited Loading

Choose a reason for hiding this comment

dongjoon-hyun left a comment

Choose a reason for hiding this comment

dongjoon-hyun commented Feb 8, 2019

gatorsmile Dec 23, 2019

Choose a reason for hiding this comment

gatorsmile Dec 23, 2019

Choose a reason for hiding this comment

Ngone51 Dec 23, 2019

Choose a reason for hiding this comment

camper42 commented Feb 5, 2019 •

edited

Loading

gaborgsomogyi commented Feb 6, 2019 •

edited

Loading

HeartSaVioR Feb 8, 2019 •

edited

Loading