[SPARK-25550][WEBUI]Environment Page of Spark Job History UI showing wrong value for spark.ui.retainedJobs

[shivusondur]

What changes were proposed in this pull request?

Currently, in History spark UI, the number of Jobs in the jobs page and "spark.ui.retainedJobs" value in
the Environment page is not matching.
So we need to override the spark properties from the History spark-default.conf properties, which will match the number of jobs showing in jobs page and value of "spark.ui.retainedJobs" in the Environment Page.
Same applies to other History specific parameters such as "spark.ui.retainedTasks" etc.

How was this patch tested?

Tested in master branch

1. In the spark-default.conf mentioned the spark.ui.retainedJobs=200
2. In spark shell command mentioned the 100 like below command
   "spark-shell --master yarn --conf spark.ui.retainedJobs=100"

Attached the test result screenshot
spark.ui.retainedJobs is showing specific to historypage

Now the total jobs showing and the spark.ui,retainedJobs value in the Environment page are matching.

[shivusondur]

@vanzin Please review this patch.

[vanzin]

Hmmm, not sure about this. That page is supposed to show the application's configuration, not the history server configuration.

If you want to add a new page to show SHS-specific configs, then that would be a better change. Otherwise this can be misleading. e.g. knowing the original config value can help someone debug things if the app ran into OOM issues.

Otherwise I don't think this should go in.

[shivusondur]

@vanzin
I modified according to your suggestion and updated the code.
Now History properties will come as seperate tab in the EnvironmentPage as shown below snap.

History UI

Only in the history UI, History Properties will be displayed. In live UI history properties won't be there

Live UI

[vanzin]

That has the same problem as before. This page is for the application, not for the history server. It should not show different things just because you're looking at it in the history server.

If you want to expose the SHS config you have to do it in a separate page that is not part of the application-specific UI. And then you have to be careful with what you show there, since the SHS config may have sensitive information that users should not see.

[AmplabJenkins]

Can one of the admins verify this patch?