[SPARK-32446][CORE] Add percentile distribution REST API & UI of peak memory metrics for all executors #34695

AngersZhuuuu · 2021-11-24T05:28:51Z

What changes were proposed in this pull request?

This pr continue the work of #29247 since origin author didn't reply for a long time.
Will add as co-author.

For the whole process of application, user may want to know each executor's peak memory usage to see the Resource utilization. The distribution of all executor's peak memory metrics usage can help users know whether or not there is a skew/bottleneck among executor resource utilization in a given stage.

We define activeOnly and quantiles query parameter in the REST API for all executors peak memory metrics distribution:

applications/<application_id>/<application_attempt/executorPeakMemoryMetricsDistribution?activeOnly=[true (default) | false]&quantiles=0.05,0.25,0.5,0.75,0.95

withSummaries: default is false, define whether to show current stage's taskMetricsDistribution and executorMetricsDistribution
quantiles: default is 0.0,0.25,0.5,0.75,1.0 only effect when withSummaries=true, it define the quantiles we use when calculating metrics distributions.

When withSummaries=true, both task metrics in percentile distribution and executor metrics in percentile distribution are included in the REST API output. The default value of withSummaries is false, i.e. no metrics percentile distribution will be included in the REST API output.

Why are the changes needed?

Always user care about executor peak usage distribution, this pr help users understand executor peak memory metrics distributions.

Does this PR introduce any user-facing change?

User can use below restful API to get all executor's peak memory metrics distribution:

applications/<application_id>/<application_attempt>/executorPeakMemoryMetricsDistribution

How was this patch tested?

…y metrics for all executors

AngersZhuuuu · 2021-11-24T05:30:33Z

@sarutak @Patil Hi, I am trying to build this pr's UI table, but I found when I write such

$('#executorSummaryMetricsTitle').html("Summary Metrics for " + "<a href='#executorsTitle'>" + allExecCnt + " Executors" + "</a>");
      $('#executorsTitle').html("Executors (" + allExecCnt + ")");

It won't change the html content. I want to know how can I enable this? Hope for your suggestion.

AngersZhuuuu · 2021-11-24T05:32:12Z

Also ping @pgandhi999

SparkQA · 2021-11-24T06:54:22Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50035/

SparkQA · 2021-11-24T07:56:59Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50035/

SparkQA · 2021-11-24T08:41:20Z

Test build #145563 has finished for PR 34695 at commit fa685bf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

pgandhi999 · 2021-11-25T22:21:09Z

I shall definitely review the PR, thank you.

pgandhi999 · 2021-11-30T18:07:25Z

Could you please post screenshots of the UI that was tested with your code changes in the PR description? Thank you.

core/src/main/resources/org/apache/spark/ui/static/executorspage.js

pgandhi999 · 2021-11-30T18:24:28Z

core/src/main/resources/org/apache/spark/ui/static/executorspage.js

+      return "Mapped Pool Memory";
+
+    case "ProcessTreeJVMVMemory":
+      return "Process Tree JVM Memory";


This should be VMemory

But there are ProcessTreeJVMVMemory and ProcessTreePythonVMemory

I am talking about the column heading, it should be Process Tree JVM VMemory.

I am talking about the column heading, it should be Process Tree JVM VMemory.

Hmmm, sorry for my mistake...

I am talking about the column heading, it should be Process Tree JVM VMemory.

updated

AngersZhuuuu · 2021-12-07T10:08:44Z

Could you please post screenshots of the UI that was tested with your code changes in the PR description? Thank you.

Done

SparkQA · 2021-12-07T10:55:08Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50461/

SparkQA · 2021-12-07T10:56:30Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50460/

SparkQA · 2021-12-07T11:38:04Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50461/

SparkQA · 2021-12-07T11:56:21Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50460/

SparkQA · 2021-12-07T12:45:46Z

Test build #145985 has finished for PR 34695 at commit b01750e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-07T13:25:15Z

Test build #145987 has finished for PR 34695 at commit b2a4289.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-08T03:37:22Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50468/

SparkQA · 2021-12-08T04:36:06Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50468/

SparkQA · 2021-12-08T05:24:21Z

Test build #145992 has finished for PR 34695 at commit 5e12e0c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang · 2022-01-06T08:56:18Z

I feel that there are too many metrics in the new table..I am using external monitor but I can't see the details of each executor when I open the executor page:

shall we remove/hide some minor metrics?

AngersZhuuuu · 2022-01-06T10:15:08Z

I feel that there are too many metrics in the new table..I am using external monitor but I can't see the details of each executor when I open the executor page:

shall we remove/hide some minor metrics?

How about just collapse this table by default? and support click to expand this?

gengliangwang · 2022-01-06T10:29:36Z

How about just collapse this table by default? and support click to expand this?

Still, the table is quite long after expanding..
Shall we hide metrics in the following checkboxes(we can include GC count as well):

Many of the metrics are already there, and the UI doesn't show them by default

AngersZhuuuu · 2022-01-07T06:03:38Z

@gengliangwang Have updated, I think current code can match your requirement.

AngersZhuuuu · 2022-01-10T03:53:56Z

Gentle ping @gengliangwang

AngersZhuuuu · 2022-01-17T02:09:43Z

ping @gengliangwang

gengliangwang · 2022-01-20T13:53:35Z

@AngersZhuuuu I will check this one in the weekend.

gengliangwang · 2022-01-24T02:53:54Z

@AngersZhuuuu shall we hide this by default if no additional metrics is selected?

gengliangwang · 2022-01-24T02:55:18Z

cc @rednaxelafx @jasonli-db as well

AngersZhuuuu · 2022-01-25T06:26:28Z

@AngersZhuuuu shall we hide this by default if no additional metrics is selected?

Updated

core/src/main/resources/org/apache/spark/ui/static/executorspage.js

gengliangwang · 2022-01-25T14:25:27Z

core/src/main/resources/org/apache/spark/ui/static/executorspage.js

 }

+function getColumnNameForExecutorMetricSummary(columnKey) {
+  switch(columnKey) {


Shall we use a string map instead of switch?

Shall we use a string map instead of switch?

Hmm, same method from stagingspage.js

Shall we use a string map instead of switch?

Updated

gengliangwang · 2022-01-25T15:06:26Z

core/src/main/resources/org/apache/spark/ui/static/executorspage.js

              sumCol.visible(!sumCol.visible());
            }
+            var para = thisBox.attr('exec-sum-idx');
+            if(para != '') {


gengliangwang · 2022-01-25T15:07:56Z

@AngersZhuuuu this LGTM overall.
My only concern is whether we need to all these metrics. cc @rednaxelafx

AngersZhuuuu · 2022-02-09T14:22:26Z

@gengliangwang No reply for a long time....How about merge first then got reply form users?

AngersZhuuuu · 2022-03-03T13:45:48Z

Gentle ping

AngersZhuuuu · 2022-03-29T03:17:15Z

gentle ping @rednaxelafx @jasonli-db

github-actions · 2022-09-29T00:32:05Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

AngersZhuuuu added 2 commits November 23, 2021 17:58

[SPARK-32446][SHS] Add percentile distribution REST API of peak memor…

3bacabd

…y metrics for all executors

Add UI stelp 1

fa685bf

github-actions bot added CORE DOCS WEB UI labels Nov 24, 2021

pgandhi999 suggested changes Nov 30, 2021

View reviewed changes

AngersZhuuuu added 5 commits December 7, 2021 16:34

Merge branch 'master' into SPARK-32446

a51b4e8

Update executorspage.js

27ab858

Update executorspage.js

b01750e

Update executorspage.js

d74f909

Update executorspage.js

b2a4289

AngersZhuuuu changed the title ~~[WIP][SPARK-32446][CORE] Add percentile distribution REST API & UI of peak memory metrics for all executors~~ [SPARK-32446][CORE] Add percentile distribution REST API & UI of peak memory metrics for all executors Dec 7, 2021

Fix JS format

5e12e0c

Merge branch 'master' into SPARK-32446

247985b

AngersZhuuuu added 3 commits January 7, 2022 11:33

Update executorspage.js

8ca2156

Update executorspage.js

1f276df

Update executorspage.js

343e178

AngersZhuuuu added 2 commits January 7, 2022 15:03

Update executorspage.js

c7b5ad8

trigger

13d0981

AngersZhuuuu added 2 commits January 25, 2022 10:16

Merge branch 'master' into SPARK-32446

c567c57

Update executorspage.js

edec1a5

gengliangwang reviewed Jan 25, 2022

View reviewed changes

core/src/main/resources/org/apache/spark/ui/static/executorspage.js Show resolved Hide resolved

gengliangwang reviewed Jan 25, 2022

View reviewed changes

Update executorspage.js

98c1ec7

gengliangwang reviewed Jan 25, 2022

View reviewed changes

Update executorspage.js

2ef92fd

github-actions bot added the Stale label Sep 29, 2022

github-actions bot closed this Sep 30, 2022

[SPARK-32446][CORE] Add percentile distribution REST API & UI of peak memory metrics for all executors #34695

[SPARK-32446][CORE] Add percentile distribution REST API & UI of peak memory metrics for all executors #34695

Uh oh!

Conversation

AngersZhuuuu commented Nov 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

AngersZhuuuu commented Nov 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AngersZhuuuu commented Nov 24, 2021

Uh oh!

SparkQA commented Nov 24, 2021

Uh oh!

SparkQA commented Nov 24, 2021

Uh oh!

SparkQA commented Nov 24, 2021

Uh oh!

pgandhi999 commented Nov 25, 2021

Uh oh!

pgandhi999 commented Nov 30, 2021

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuu commented Dec 7, 2021

Uh oh!

SparkQA commented Dec 7, 2021

Uh oh!

SparkQA commented Dec 7, 2021

Uh oh!

SparkQA commented Dec 7, 2021

Uh oh!

SparkQA commented Dec 7, 2021

Uh oh!

SparkQA commented Dec 7, 2021

Uh oh!

SparkQA commented Dec 7, 2021

Uh oh!

SparkQA commented Dec 8, 2021

Uh oh!

SparkQA commented Dec 8, 2021

Uh oh!

SparkQA commented Dec 8, 2021

Uh oh!

gengliangwang commented Jan 6, 2022

Uh oh!

AngersZhuuuu commented Jan 6, 2022

Uh oh!

gengliangwang commented Jan 6, 2022

Uh oh!

AngersZhuuuu commented Jan 7, 2022

Uh oh!

AngersZhuuuu commented Jan 10, 2022

Uh oh!

AngersZhuuuu commented Jan 17, 2022

Uh oh!

gengliangwang commented Jan 20, 2022

Uh oh!

gengliangwang commented Jan 24, 2022

Uh oh!

gengliangwang commented Jan 24, 2022

Uh oh!

AngersZhuuuu commented Jan 25, 2022

Uh oh!

Uh oh!

Choose a reason for hiding this comment

AngersZhuuuu commented Nov 24, 2021 •

edited

Loading

AngersZhuuuu commented Nov 24, 2021 •

edited

Loading