[SPARK-11051][Core] Do not allow local checkpointing after the RDD is materialized and checkpointed

[viirya]

JIRA: https://issues.apache.org/jira/browse/SPARK-11051

When a RDD is materialized and checkpointed, its partitions and dependencies are cleared. If we allow local checkpointing on it and assign LocalRDDCheckpointData to its checkpointData. Next time when the RDD is materialized again, the error will be thrown.

[SparkQA]

Test build #43560 has finished for PR 9072 at commit 124aab0.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

I suspect this is a good restriction, but cc @tdas regarding the semantics.

I feel like this code construct is getting hard to read. I don't feel strongly, but is if (checkpointData.isDefined && isCheckpointed) not simpler? then there are just two branches, one return point.

[viirya]

In addition to checkpointData.isDefined && isCheckpointed, we still need to check isLocallyCheckpointed, because when checkpointData.isDefined is true, it is possibly a local checkpointing.

[viirya]

retest this please.

[SparkQA]

Test build #43565 has finished for PR 9072 at commit 124aab0.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

retest this please.

[SparkQA]

Test build #43566 has finished for PR 9072 at commit 124aab0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tdas]

Actually, best for @andrewor14 to take a look, who implemented local checkpointing and has more context.

[andrewor14]

can you rename this to isCheckpointedAndMaterialized and update the java docs? Right now it's a little confusing because it can be interpreted as "marked for checkpointing" only, which is actually what the existing java docs say.

[andrewor14]

@viirya thanks for the fix. I suggested an alternative which I think is a little clearer. Could you also add a regression test for this in CheckpointSuite?

[viirya]

@andrewor14 Thanks. I will update this patch and add the test later today.

[andrewor14]

retest this please

[andrewor14]

this should be private[spark]

[SparkQA]

Test build #43813 has finished for PR 9072 at commit 448774f.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #43829 has finished for PR 9072 at commit dc58498.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

why add this? I don't think this is used except in tests

[andrewor14]

@viirya this looks pretty close. I don't think we need to add the method in Java. Once you remove that and update the java docs I will merge this. Thanks for fixing this issue.

[viirya]

@andrewor14 Thanks. I've updated this. Please see if there is any problem.

[SparkQA]

Test build #43873 has finished for PR 9072 at commit 3958bc2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

ping @andrewor14 any other comments?

[andrewor14]

Yup LGTM I'm merging this into master 1.5. Thanks for fixing this @viirya !