[SPARK-8374] [YARN] Job frequently hangs after YARN preemption

[xuchenCN]

Issue description SPARK-8374

Application starve because YARN scheduler preemption.
This PR fix two issues

1. Application starve because CoarseGrainedSchedulerBackend.removeExecutor was happened before HeartbeatReceiver.killExecutorThread

2.SpakrContext.killExecutor less executors than before BUT HeartbeatReceiver just want to expire the executor and expect new one

[xuchenCN]

CC @sryza

[andrewor14]

ok to test

[SparkQA]

Test build #36219 has finished for PR 7128 at commit c1fa754.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Heartbeat(workerId: String, worker: RpcEndpointRef) extends DeployMessage
  • case class RegisteredWorker(master: RpcEndpointRef, masterWebUiUrl: String) extends DeployMessage
  • case class RegisterApplication(appDescription: ApplicationDescription, driver: RpcEndpointRef)
  • case class RegisteredApplication(appId: String, master: RpcEndpointRef) extends DeployMessage
  • case class SubmitDriverResponse(
  • case class KillDriverResponse(
  • case class MasterChanged(master: RpcEndpointRef, masterWebUiUrl: String)

[AmplabJenkins]

Can one of the admins verify this patch?

[andrewor14]

This is a confusing name. Without reading the java docs I have no idea what the difference between killExecutor and expireExecutor is

[andrewor14]

@xuchenCN The issue this patch is targeted for is a duplicate of SPARK-8119. There is already a fix for that in #7107. I believe the changes there are simpler and we should move forward with that one instead. Would you mind closing this PR?

[xuchenCN]

@andrewor14 All right close this PR

[tgravescs]

@xuchenCN can you please close this then