Skip to content

[SPARK-22860] [SPARK-24621] [Core] [WebUI] - hide key password from linux ps listing and masterwebui point to https if ssl enabled #21514

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tooptoop4 wants to merge 7 commits into from
Closed

[SPARK-22860] [SPARK-24621] [Core] [WebUI] - hide key password from linux ps listing and masterwebui point to https if ssl enabled #21514

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
380d97d
[SPARK-22860] [Core] - hide key password from linux ps listing
Jun 8, 2018
f38de2b
SPARK-24621
Jun 30, 2018
825380a
updated according to review
Jul 1, 2018
050b226
style guide <100 char line of code
Jul 16, 2018
bc8a833
added UT and changed 4 lines to 1 line
Jul 16, 2018
39bb2a0
style comments
Jul 17, 2018
d4de123
nit pickin'
Jul 17, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion core/src/main/scala/org/apache/spark/deploy/master/Master.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,9 @@ private[deploy] class Master(
logInfo(s"Running Spark version ${org.apache.spark.SPARK_VERSION}")
webUi = new MasterWebUI(this, webUiPort)
webUi.bind()
masterWebUiUrl = "http://" + masterPublicAddress + ":" + webUi.boundPort
val sslEnabled = conf.getBoolean("spark.ssl.enabled", false)
val uriScheme = if (sslEnabled) { "https://" } else { "http://" }
masterWebUiUrl = uriScheme + masterPublicAddress + ":" + webUi.boundPort
if (reverseProxy) {
masterWebUiUrl = conf.get("spark.ui.reverseProxyUrl", masterWebUiUrl)
webUi.addProxy()
Expand Down
7 changes: 5 additions & 2 deletions core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,8 +99,11 @@ private[spark] class StandaloneSchedulerBackend(
// Start executors with a few necessary configs for registering with the scheduler
val sparkJavaOpts = Utils.sparkJavaOpts(conf, SparkConf.isExecutorStartupConf)
val javaOpts = sparkJavaOpts ++ extraJavaOpts
val command = Command("org.apache.spark.executor.CoarseGrainedExecutorBackend",
args, sc.executorEnvs, classPathEntries ++ testingClassPath, libraryPathEntries, javaOpts)
val javaOptsFiltered = javaOpts.filterNot { opt =>
opt.startsWith("-Dspark.ssl.keyStorePassword") || opt.startsWith("-Dspark.ssl.keyPassword")
Copy link
Contributor

@mgaido91 mgaido91 Jul 17, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wrong indentation: missing 2 spaces.

}
val command = Command("org.apache.spark.executor.CoarseGrainedExecutorBackend", args,
sc.executorEnvs, classPathEntries ++ testingClassPath, libraryPathEntries, javaOptsFiltered)
Copy link
Contributor

@mgaido91 mgaido91 Jul 17, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wrong indentation here too, missing 2 spaces. Moreover, in such cases, we usually put one argument per line, so:

val command = Command("org.apache.spark.executor.CoarseGrainedExecutorBackend",
  args,
  sc.executorEnvs,
  ...)

val webUrl = sc.ui.map(_.webUrl).getOrElse("")
val coresPerExecutor = conf.getOption("spark.executor.cores").map(_.toInt)
// If we're using dynamic allocation, set our initial executor limit to 0 for now.
Expand Down
34 changes: 34 additions & 0 deletions core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -282,6 +282,40 @@ class MasterSuite extends SparkFunSuite
}
}

test("SPARK-24621: https urls when ssl enabled") {
implicit val formats = org.json4s.DefaultFormats
val conf = new SparkConf()
conf.set("spark.ssl.enabled", "true")
val localCluster = new LocalSparkCluster(2, 2, 512, conf)
localCluster.start()
try {
eventually(timeout(5 seconds), interval(100 milliseconds)) {
val json = Source.fromURL(s"https://localhost:${localCluster.masterWebUIPort}/json")
.getLines().mkString("\n")
assert(json.contains('<a href="https://'))
Copy link
Contributor

@mgaido91 mgaido91 Jul 17, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

" instead of '

}
} finally {
localCluster.stop()
}
}

test("SPARK-24621: http urls when ssl disabled") {
implicit val formats = org.json4s.DefaultFormats
val conf = new SparkConf()
conf.set("spark.ssl.enabled", "false")
val localCluster = new LocalSparkCluster(2, 2, 512, conf)
localCluster.start()
try {
eventually(timeout(5 seconds), interval(100 milliseconds)) {
val json = Source.fromURL(s"http://localhost:${localCluster.masterWebUIPort}/json")
.getLines().mkString("\n")
assert(!json.contains('<a href="https://'))
Copy link
Contributor

@mgaido91 mgaido91 Jul 17, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

}
} finally {
localCluster.stop()
}
}

test("master/worker web ui available with reverseProxy") {
implicit val formats = org.json4s.DefaultFormats
val reverseProxyUrl = "http://localhost:8080"
Expand Down