From d170730ffa1f98d4b117625b1870f46a7da3e241 Mon Sep 17 00:00:00 2001 From: Marcelo Vanzin Date: Thu, 17 Sep 2015 16:05:21 -0700 Subject: [PATCH 1/5] [SPARK-10676] [docs] Add documentation for SASL encryption options. --- docs/security.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/docs/security.md b/docs/security.md index d4ffa60e59a3..0281797fc39e 100644 --- a/docs/security.md +++ b/docs/security.md @@ -23,7 +23,8 @@ If your applications are using event logging, the directory where the event logs ## Encryption -Spark supports SSL for Akka and HTTP (for broadcast and file server) protocols. However SSL is not supported yet for WebUI and block transfer service. +Spark supports SSL for Akka and HTTP (for broadcast and file server) protocols. SASL encryption is +supported for the block transfer service. Encryption is not yet supported for the WebUI. Connection encryption (SSL) configuration is organized hierarchically. The user can configure the default SSL settings which will be used for all the supported communication protocols unless they are overwritten by protocol-specific settings. This way the user can easily provide the common settings for all the protocols without disabling the ability to configure each one individually. The common SSL settings are at `spark.ssl` namespace in Spark configuration, while Akka SSL configuration is at `spark.ssl.akka` and HTTP for broadcast and file server SSL configuration is at `spark.ssl.fs`. The full breakdown can be found on the [configuration page](configuration.html). @@ -47,6 +48,17 @@ follows: * Import all exported public keys into a single trust-store * Distribute the trust-store over the nodes +### Configuring SASL Encryption + +To enable SASL encryption for an application, set `spark.authenticate.enableSaslEncryption` to +`true`. Authentication (`spark.authenticate`) must also be turned on for this option to take +effect. + +When using an external shuffle service, it's possible to disable unencrypted connections by setting +`spark.network.sasl.serverAlwaysEncrypt` to `true` in the shuffle service's configuration. If that +option is enabled, applications that are not set up to use SASL encryption will fail to connect to +the shuffle service. + ## Configuring Ports for Network Security Spark makes heavy use of the network, and some environments have strict requirements for using tight From f2e95153a52175113cdf39fffa3fe0b232f2cb79 Mon Sep 17 00:00:00 2001 From: Marcelo Vanzin Date: Thu, 17 Sep 2015 16:34:17 -0700 Subject: [PATCH 2/5] Feedback. --- docs/security.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/security.md b/docs/security.md index 0281797fc39e..043b0d18d85a 100644 --- a/docs/security.md +++ b/docs/security.md @@ -26,6 +26,10 @@ If your applications are using event logging, the directory where the event logs Spark supports SSL for Akka and HTTP (for broadcast and file server) protocols. SASL encryption is supported for the block transfer service. Encryption is not yet supported for the WebUI. +Encryption is not yet supported for data stored by Spark in temporary local storage, such as shuffle +files, cached data, and other application files. If encrypting this data is desired, a workaround is +to configure your cluster manager to store application data on encrypted disks. + Connection encryption (SSL) configuration is organized hierarchically. The user can configure the default SSL settings which will be used for all the supported communication protocols unless they are overwritten by protocol-specific settings. This way the user can easily provide the common settings for all the protocols without disabling the ability to configure each one individually. The common SSL settings are at `spark.ssl` namespace in Spark configuration, while Akka SSL configuration is at `spark.ssl.akka` and HTTP for broadcast and file server SSL configuration is at `spark.ssl.fs`. The full breakdown can be found on the [configuration page](configuration.html). SSL must be configured on each node and configured for each component involved in communication using the particular protocol. @@ -50,9 +54,9 @@ follows: ### Configuring SASL Encryption -To enable SASL encryption for an application, set `spark.authenticate.enableSaslEncryption` to -`true`. Authentication (`spark.authenticate`) must also be turned on for this option to take -effect. +SASL encryption is currently supported for the block transfer service when authentication +(`spark.authenticate`) is enabled. To enable SASL encryption for an application, set +`spark.authenticate.enableSaslEncryption` to `true` in the application's configuration. When using an external shuffle service, it's possible to disable unencrypted connections by setting `spark.network.sasl.serverAlwaysEncrypt` to `true` in the shuffle service's configuration. If that From 9fb59ee404e00da760e56b2b451e9adc1dc74130 Mon Sep 17 00:00:00 2001 From: Marcelo Vanzin Date: Thu, 17 Sep 2015 19:22:34 -0700 Subject: [PATCH 3/5] Add header to separate SSL text from rest. --- docs/security.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/security.md b/docs/security.md index 043b0d18d85a..36f581732c00 100644 --- a/docs/security.md +++ b/docs/security.md @@ -30,6 +30,8 @@ Encryption is not yet supported for data stored by Spark in temporary local stor files, cached data, and other application files. If encrypting this data is desired, a workaround is to configure your cluster manager to store application data on encrypted disks. +### SSL Configuration + Connection encryption (SSL) configuration is organized hierarchically. The user can configure the default SSL settings which will be used for all the supported communication protocols unless they are overwritten by protocol-specific settings. This way the user can easily provide the common settings for all the protocols without disabling the ability to configure each one individually. The common SSL settings are at `spark.ssl` namespace in Spark configuration, while Akka SSL configuration is at `spark.ssl.akka` and HTTP for broadcast and file server SSL configuration is at `spark.ssl.fs`. The full breakdown can be found on the [configuration page](configuration.html). SSL must be configured on each node and configured for each component involved in communication using the particular protocol. From 8db3b07fc22ecf28294f3ae524d0d8720b3c2b88 Mon Sep 17 00:00:00 2001 From: Marcelo Vanzin Date: Thu, 17 Sep 2015 19:23:43 -0700 Subject: [PATCH 4/5] Slight reword of SSL text. --- docs/security.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/security.md b/docs/security.md index 36f581732c00..177109415180 100644 --- a/docs/security.md +++ b/docs/security.md @@ -32,7 +32,7 @@ to configure your cluster manager to store application data on encrypted disks. ### SSL Configuration -Connection encryption (SSL) configuration is organized hierarchically. The user can configure the default SSL settings which will be used for all the supported communication protocols unless they are overwritten by protocol-specific settings. This way the user can easily provide the common settings for all the protocols without disabling the ability to configure each one individually. The common SSL settings are at `spark.ssl` namespace in Spark configuration, while Akka SSL configuration is at `spark.ssl.akka` and HTTP for broadcast and file server SSL configuration is at `spark.ssl.fs`. The full breakdown can be found on the [configuration page](configuration.html). +Configuration for SSL is organized hierarchically. The user can configure the default SSL settings which will be used for all the supported communication protocols unless they are overwritten by protocol-specific settings. This way the user can easily provide the common settings for all the protocols without disabling the ability to configure each one individually. The common SSL settings are at `spark.ssl` namespace in Spark configuration, while Akka SSL configuration is at `spark.ssl.akka` and HTTP for broadcast and file server SSL configuration is at `spark.ssl.fs`. The full breakdown can be found on the [configuration page](configuration.html). SSL must be configured on each node and configured for each component involved in communication using the particular protocol. From f656dfa367e3aeb7e54787015596532bae40a15a Mon Sep 17 00:00:00 2001 From: Marcelo Vanzin Date: Fri, 18 Sep 2015 16:06:58 -0700 Subject: [PATCH 5/5] Update configuration.md. --- docs/configuration.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/configuration.md b/docs/configuration.md index 1a701f18881f..f9452a90e0eb 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1303,6 +1303,22 @@ Apart from these, the following properties are also available, and may be useful not running on YARN and authentication is enabled. + + spark.authenticate.enableSaslEncryption + false + + Enable encrypted communication when authentication is enabled. This option is currently + only supported by the block transfer service. + + + + spark.network.sasl.serverAlwaysEncrypt + false + + Disable unencrypted connections for services that support SASL authentication. This is + currently supported by the external shuffle service. + + spark.core.connection.ack.wait.timeout 60s