[SPARK-5771] Master UI inconsistently displays application cores

[andrewor14]

If the user calls sc.stop(), then the number of cores under "Completed Applications" will be 0. If the user does not call sc.stop(), then the number of cores will be however many cores were being used before the application exited. This PR makes both cases have the behavior of the latter.

Note that there have been a series of PR that attempted to fix this. For the full discussion, please refer to #4841. The unregister event is necessary because of a subtle race condition explained in that PR.

Tested this locally with and without calling sc.stop().

[SparkQA]

Test build #29130 has started for PR 5177 at commit 62449d1.

• This patch merges cleanly.

[andrewor14]

@jerryshao @srowen @marsishandsome

[SparkQA]

Test build #29130 has finished for PR 5177 at commit 62449d1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class UnregisterApplication(appId: String)
  • case class ExecutorCacheTaskLocation(override val host: String, executorId: String)
  • class MLPairRDDFunctions[K: ClassTag, V: ClassTag](self: RDD[(K, V)]) extends Serializable
  • class NaiveBayesModel(Saveable, Loader):
  • class SqlParser extends AbstractSparkSQLParser with DataTypeParser
  • implicit class DslSymbol(sym: Symbol) extends ImplicitAttribute
  • case class CombineSum(child: Expression) extends AggregateExpression
  • case class CombineSumFunction(expr: Expression, base: AggregateExpression)
  • case class Alias(child: Expression, name: String)(
  • protected[sql] class DataTypeException(message: String) extends Exception(message)
  • class Column(protected[sql] val expr: Expression) extends Logging

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29130/
Test PASSed.

[jerryshao]

these changes looks good to me :)

[andrewor14]

LGTM2 I'm merging this into master thanks guys!