[SPARK-11253][SQL] reset all accumulators in physical operators before execute an action

[cloud-fan]

With this change, our query execution listener can get the metrics correctly.

The UI still looks good after this change.

[rxin]

() since it has side effect

[SparkQA]

Test build #44126 has finished for PR 9215 at commit 4589e66.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44129 has finished for PR 9215 at commit 6397cf5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

Can we run the same plan physical plan multiple times to make sure metrics are good?

[yhuai]

Can you also double check if our ui looks good after this change?

[SparkQA]

Test build #44186 has finished for PR 9215 at commit 778992e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

@cloud-fan sorry. I just realized that this method is in the critical path (when we calculate numRows). How about we remove this change and document it clear that those negative initial values will have a small impact on the sum of memory consumption?

[cloud-fan]

I was trying to reuse current codebase and didn't create a new SQLMetric(including new MetricValue, MetricParam, etc.). Is it worth to create a new one so that we won't hurt performance for numRows?

[yhuai]

I guess it is probably not worth that right now. The impact of those -1 just too small (1048576 tasks for 1 MB). I think we can do that later. What do you think?

[SparkQA]

Test build #44237 has finished for PR 9215 at commit 4ff8912.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44319 has finished for PR 9215 at commit b088c70.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

Sorry. Just one last thing. I think we need to call unregister at the end of every test, right?

[cloud-fan]

good catch!

[SparkQA]

Test build #44332 has finished for PR 9215 at commit 6923c4f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

Thanks! Merging to master.

[scwf]

should this merged to branch-1.5?

[yhuai]

@scwf It will not be merged to branch-1.5 because our UI handles those metric updates correctly. This problem was exposed after we call the QueryExecutionListener logic (see #9078), which is a 1.6 feature..