[SPARK-17171][WEB UI] DAG will list all partitions in the graph

[cenyuhai]

What changes were proposed in this pull request?

DAG will list all partitions in the graph, it is too slow and hard to see all graph.
Always we don't want to see all partitions，we just want to see the relations of DAG graph.
So I just show 2 root nodes for Rdds.

Before this PR, the DAG graph looks like dag1.png, dag3.png, after this PR, the DAG graph looks like dag2.png,dag4.png

[SparkQA]

Test build #64158 has finished for PR 14737 at commit 595453f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

The whole if-else is just ids.isEmpty || parentIds.exists(id => ids.contains(id) || !dropRDDIds.contains(id)) ?

[cenyuhai]

yes, you are right...

[SparkQA]

Test build #64167 has finished for PR 14737 at commit 770b3de.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cenyuhai]

@srowen

[markhamstra]

NODES, both here and in spark.ui.retainedNodes if far too ambiguous and non-specific for this configuration value -- "node" is already overloaded too many times in the existing Spark code and documentation; we don't need or want to add another overload.

Additionally, the default behavior should be the same as current behavior, since the change in behavior would be unexpected and it is far from clear to me that the overwhelming majority of users would prefer the proposed new behavior.

[SparkQA]

Test build #64173 has finished for PR 14737 at commit 4019842.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cenyuhai]

@srowen please review the latest codes, thank you.

[srowen]

Pardon the dumb question, but your before and after pictures seem like different graphs. Is that really showing the before and after of one display?

[cenyuhai]

I am very sorry about, the first picture is for stage, the second picture is for job, but it is the same job "select count(1) from partitionedTables ". I upload new pictures. dag1.png vs dag4.png, dag3.png vs dag2.png.

[srowen]

I can see the value in this, but it also removes any indication that there are more elements to the graph that are not displayed. I wonder if there is any easy way to do that or do I misunderstand? Also why 2, just arbitrary?

[cenyuhai]

@srowen Why I set this value 2, because a "JOIN" action needs 2 elements.Users want to know the relations from DAG graphs, if there are too manys elements, it is meesy. I have changed the codes as @markhamstra said, it will not remove elements by default. Users can set this value as they like. But I recommend that 2 is better.

[srowen]

OK, another config eh. If there's no default change in behavior, and it's undocumented and it doesn't introduce much complexity, seems OK

[cenyuhai]

@srowen If it is ok，can you merge this pr to master？thank you.

[srowen]

Minor stuff like:

if (rdd.parnetIds.isEmpty) {
  rootNoteCount += 1
}

[SparkQA]

Test build #65152 has finished for PR 14737 at commit 40def0f.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #65190 has finished for PR 14737 at commit 3c72dde.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Now we're just down to style stuff but I how about...

val isAllowed =
  if (parentId.isEmpty) {
    rootNodeCount += 1
    rootNodeCount <= retainedNodes
  } else {
    parentIds.exists(...)
  }

[SparkQA]

Test build #65209 has finished for PR 14737 at commit 5163a51.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

I looked one more time and the only thing that crossed my mind is whether spark.ui.dagGraph.retainedRootRDDs is the rightest naming convention, but, it's a hidden internal property at the moment, and I don't have a better idea, so let's leave it.

[srowen]

Merged to master

[cenyuhai]

thank you!