Skip to content

[SPARK-3410] The priority of shutdownhook for ApplicationMaster should not be integer literal #2283

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sarutak wants to merge 8 commits into from
Closed

[SPARK-3410] The priority of shutdownhook for ApplicationMaster should not be integer literal #2283

Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
717aba2
Modified ApplicationMaster to make to keep the priority of shutdown h…
sarutak Sep 5, 2014
217d1a4
Removed unused import statements
sarutak Sep 5, 2014
4c5cb93
Modified the priority for AM's shutdown hook
sarutak Sep 5, 2014
2f0aee3
Merge branch 'master' of git://git.apache.org/spark into SPARK-3410
sarutak Sep 12, 2014
54eb68f
Changed Shutdown hook priority to 20
sarutak Sep 13, 2014
ee6f1aa
Added constant "SHUTDOWN_HOOK_PRIORITY" to ApplicationMaster
sarutak Sep 13, 2014
bd6cc53
Modified style
sarutak Sep 13, 2014
1d44fef
Merge branch 'master' of git://git.apache.org/spark into SPARK-3410
sarutak Sep 14, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,8 @@ import java.io.IOException
import java.net.Socket
import java.util.concurrent.atomic.AtomicReference

import scala.collection.JavaConversions._
import scala.util.Try

import akka.actor._
import akka.remote._
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileSystem, Path}
import org.apache.hadoop.util.ShutdownHookManager
import org.apache.hadoop.yarn.api._
Expand Down Expand Up @@ -107,8 +103,11 @@ private[spark] class ApplicationMaster(args: ApplicationMasterArguments,
}
}
}
// Use priority 30 as it's higher than HDFS. It's the same priority MapReduce is using.
ShutdownHookManager.get().addShutdownHook(cleanupHook, 30)

// Use higher priority than FileSystem.
assert(ApplicationMaster.SHUTDOWN_HOOK_PRIORITY > FileSystem.SHUTDOWN_HOOK_PRIORITY)
ShutdownHookManager
.get().addShutdownHook(cleanupHook, ApplicationMaster.SHUTDOWN_HOOK_PRIORITY)

// Call this to force generation of secret so it gets populated into the
// Hadoop UGI. This has to happen before the startUserClass which does a
Expand Down Expand Up @@ -407,6 +406,8 @@ private[spark] class ApplicationMaster(args: ApplicationMasterArguments,

object ApplicationMaster extends Logging {

val SHUTDOWN_HOOK_PRIORITY: Int = 30

private var master: ApplicationMaster = _

def main(args: Array[String]) = {
Expand Down