[SPARK-23257][K8S] Kerberos Support for Spark on K8S #21669

ifilonenko · 2018-06-29T08:47:21Z

What changes were proposed in this pull request?

This is the work on setting up Secure HDFS interaction with Spark-on-K8S.
The architecture is discussed in this community-wide google doc
This initiative can be broken down into 4 Stages

STAGE 1

Detecting HADOOP_CONF_DIR environmental variable and using Config Maps to store all Hadoop config files locally, while also setting HADOOP_CONF_DIR locally in the driver / executors

STAGE 2

Grabbing TGT from LTC or using keytabs+principle and creating a DT that will be mounted as a secret or using a pre-populated secret

STAGE 3

Driver

STAGE 4

Executor

How was this patch tested?

Locally tested on a single-noded, pseudo-distributed Kerberized Hadoop Cluster

E2E Integration tests [SPARK-25750][K8S][TESTS] Kerberos Support Integration Tests #22608
Unit tests

Docs and Error Handling?

Docs
Error Handling

Contribution Credit

@kimoonkim @skonto

SparkQA · 2018-06-29T09:03:05Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/583/

SparkQA · 2018-06-29T09:07:54Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/583/

SparkQA · 2018-06-29T12:17:14Z

Test build #92471 has finished for PR 21669 at commit 73f2777.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

felixcheung · 2018-06-30T05:25:40Z

btw, have you sent out this + doc to dev@spark.apache.org?

SparkQA · 2018-07-01T10:32:12Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/609/

SparkQA · 2018-07-01T10:37:28Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/609/

SparkQA · 2018-07-01T14:46:34Z

Test build #92512 has finished for PR 21669 at commit 6069be5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-02T05:02:42Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/613/

SparkQA · 2018-07-02T05:08:11Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/613/

SparkQA · 2018-07-02T07:05:01Z

Test build #92520 has finished for PR 21669 at commit 000120f.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

skonto · 2018-07-03T13:14:42Z

core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala


    // assure a keytab is available from any place in a JVM
-    if (clusterManager == YARN || clusterManager == LOCAL || isMesosClient) {
+    if (clusterManager == YARN || clusterManager == LOCAL || isMesosClient || isKubernetesCluster) {


This check has been restrictive for customers in the past. There are cases where spark submit should not have the file locally and keytab should be mounted as a secret within the cluster for example.

This check can be removed, but I included it since I believed that the keytab shouldn't be stored as a secret for security reasons and should instead be only accessible from the JVM.

skonto · 2018-07-03T13:20:32Z

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala


+  val KUBERNETES_KERBEROS_SUPPORT =
+    ConfigBuilder("spark.kubernetes.kerberos.enabled")
+      .doc("Specify whether your job is a job that will require a Delegation Token to access HDFS")


I think kerberos goes beyond DTs so it shouldnt be specific to that. Also I think you dont need the user to pass that. You just need to call: UserGroupInformation.isSecurityEnabled instead of getting that property from config.

skonto · 2018-07-03T13:29:11Z

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala

+    SPARK_HADOOP_PREFIX + "hadoop.security.authentication"
+
+  // Kerberos Token-Refresh Server
+  val KERBEROS_REFRESH_LABEL_KEY = "refresh-hadoop-tokens"


I left a comment in the design doc. Can we also provide the option for using an existing renewal service like when integrating with an external hadoop cluster where people already have that. AFAIK hadoop libs do the renewal behind the scenes by talking to the appropriate service.
This is the current implementation with mesos for integrating with existing clusters.

Because our original architecture had the opinion that the renewal service pod will exist as a separate micro-service, that option could be handled by that renewal service. We used this label to detect that this specific secret was to be renewed. But if we wished to use another renewal service via some existing service, we might be able to just grab an Array[Byte] from some DTManager that may exist in their external Hadoop clusters, and store in a secret. Thank you for this note in the design doc.

It would be good to have that option.

Agreed but that would be out of the scope of this PR as the renewal service is a separate micro-service (instead of a running thread), and thereby that logic would be housed in a separate PR governing the opinion of the renewal service pods "DT retrieving" protocol

Is this label actually being used for anything?

skonto · 2018-07-03T20:09:59Z

...kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/OptionRequirements.scala

+    }
+  }
+
+  def requireNandDefined(opt1: Option[_], opt2: Option[_], errMessage: String): Unit = {


requireNandDefined is spelling ok? Also that function is not used, should be removed or will be used later when PR will be updated?

skonto · 2018-07-03T20:31:02Z

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala

  val ENV_CLASSPATH = "SPARK_CLASSPATH"
  val ENV_DRIVER_BIND_ADDRESS = "SPARK_DRIVER_BIND_ADDRESS"
  val ENV_SPARK_CONF_DIR = "SPARK_CONF_DIR"
+  val ENV_SPARK_USER = "SPARK_USER"


I guess this is for setting the correct user. But I think hadoop libs should pick the correct user like in SparkContext where Utils.getCurrentUserName() is used. In addition I think we should allow the pods to run with any username (eg. with customized images) and we should have a security context per container like: SecurityContextBuilder().withRunAsUser(). In the latter scenario hadoop libraries will pick up that logged in user as well. A common scenario would be integrating the container with host PAM: jupyterhub/jupyterhub#535 https://medium.com/@pawitp/syncing-host-and-container-users-in-docker-39337eff0094 or with LDAP etc.
Btw if you change the user with which the container user needs also to exist on the image when basic authorization is used (unix groups).

SparkQA · 2018-07-07T03:57:40Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/743/

SparkQA · 2018-07-07T04:02:13Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/743/

SparkQA · 2018-07-07T10:28:01Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/746/

SparkQA · 2018-07-07T10:33:06Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/746/

SparkQA · 2018-07-07T14:37:04Z

Test build #92705 has finished for PR 21669 at commit 13b3adc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-18T04:46:36Z

Test build #93211 has finished for PR 21669 at commit 13b3adc.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2018-07-30T21:57:04Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1501/

SparkQA · 2018-07-30T22:05:45Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1501/

SparkQA · 2018-07-31T02:32:16Z

Test build #93807 has finished for PR 21669 at commit 0939738.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-10-09T19:00:35Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3833/

SparkQA · 2018-10-09T19:10:29Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3833/

liyinan926 · 2018-10-09T20:29:15Z

...re/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopKerberosLogin.scala

-     val initialTokenDataKeyName = s"$KERBEROS_SECRET_KEY_PREFIX-$currentTime"
-     val newSecretName =
-       s"$kubernetesResourceNamePrefix-$KERBEROS_DELEGEGATION_TOKEN_SECRET_NAME.$currentTime"
+     val initialTokenDataKeyName = KERBEROS_SECRET_KEY_PREFIX


Nit: you don't need this variable and you can rename KERBEROS_SECRET_KEY_PREFIX to KERBEROS_SECRET_KEY.

ifilonenko · 2018-10-09T20:37:15Z

thank you @liyinan926 for your review! @vanzin, ill leave it to you to give the final LGTM

vanzin

Man, it's really tiring to go through the code and keep seeing the same style issues I've pointed out many times already. Sorry if I sound frustrated, but there's so much of that I can do before I start to give up on the review.

Please go through your code and fix all of those.

vanzin · 2018-10-09T20:56:55Z

docs/running-on-kubernetes.md

  </td>
 </tr>
+<tr>
+  <td><code>spark.kubernetes.kerberos.krb5.location</code></td>


Kinda let for this one, but since this is a local path, I think krb5.path makes more sense. Or maybe krb5.confFile to contrast with the config map alternative.

vanzin · 2018-10-09T20:57:19Z

docs/running-on-kubernetes.md

+  <td><code>spark.kubernetes.kerberos.krb5.configMapName</code></td>
+  <td><code>(none)</code></td>
+  <td>
+   Specify the name of the ConfigMap, containing the krb5 file, to be mounted on the driver and executors


"krb5.conf file"

vanzin · 2018-10-09T20:59:03Z

docs/running-on-kubernetes.md

+  <td><code>spark.kubernetes.kerberos.tokenSecret.name</code></td>
+  <td><code>(none)</code></td>
+  <td>
+    Specify the name of the secret where your existing delegation token is stored. This removes the need for the job user


"tokens are"

vanzin · 2018-10-09T20:59:18Z

docs/running-on-kubernetes.md

+  <td><code>spark.kubernetes.kerberos.tokenSecret.itemKey</code></td>
+  <td><code>(none)</code></td>
+  <td>
+    Specify the item key of the data where your existing delegation token is stored. This removes the need for the job user 


"tokens are"

vanzin · 2018-10-09T21:00:02Z

docs/security.md

+`spark.kubernetes.hadoop.configMapName` as well as either
+`spark.kubernetes.kerberos.krb5.location` or `spark.kubernetes.kerberos.krb5.configMapName`.
+
+It also important to note that the KDC needs to be visible from inside the containers if the user uses a local


Not just "if the user uses a local krb5 file".

vanzin · 2018-10-09T21:39:02Z

...core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala

+        hFiles <- hadoopConfigurationFiles
+      } yield {
+       HadoopBootstrapUtil.buildHadoopConfigMap(
+         hName,


all args fit in the original line

vanzin · 2018-10-09T21:39:30Z

...core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala

+     }
+
+     // Kerberos DT Secret
+     val kerberosDTSecret = for {


really, please: kerberosConfSpec.flatMap(_.dtSecret).

To my knowledge and after a quick test in the shell, you can't use flatMap as this would cause a: dtSecret is not a member of Nothing error for one of the input cases:
i.e. None.flatMap(_.dtSecret) is the cause of the failure.
:( I understand that it isn't preferred but, I think, this might be the cleanest way to handle the cases:

None,

Some(KerberosConfSpec(dtSecret: None,...)),

Some(KerberosConfSpec(dtSecret: Some(Secret),...)

the type of kerberosConfSpec is known, no? Then it should work.

scala> case class Foo(bar: Option[String]) defined class Foo scala> scala> Some(Foo(Some("bar"))).flatMap(_.bar) res1: Option[String] = Some(bar) scala> Some(Foo(None)).flatMap(_.bar) res2: Option[String] = None scala> Option[Foo](null).flatMap(_.bar) res3: Option[String] = None

A raw "None" would not work, but you don't have a raw "None" here; you have a val kerberosConfSpec, of type Option[KerberosConfigSpec], and value None, so you have all the type information you need.

vanzin · 2018-10-09T21:41:32Z

...re/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfExecutorFeatureStep.scala

+  * This step is responsible for mounting the DT secret for the executors
+  */
+private[spark] class KerberosConfExecutorFeatureStep(
+   kubernetesConf: KubernetesConf[KubernetesExecutorSpecificConf])


arg indentation. this is another class that is all off with 3-space indentation.

vanzin · 2018-10-09T21:41:59Z

...re/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfExecutorFeatureStep.scala

+  */
+private[spark] class KerberosConfExecutorFeatureStep(
+   kubernetesConf: KubernetesConf[KubernetesExecutorSpecificConf])
+   extends KubernetesFeatureConfigStep with Logging{


space before {

vanzin · 2018-10-09T21:42:23Z

...re/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopBootstrapUtil.scala

+    * @return a modified SparkPod
+    */
+  def bootstrapKerberosPod(
+    dtSecretName: String,


arg indentation

SparkQA · 2018-10-09T22:32:13Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3835/

SparkQA · 2018-10-09T22:41:24Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3835/

…t fail

SparkQA · 2018-10-10T02:23:21Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3839/

SparkQA · 2018-10-10T02:33:31Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3839/

SparkQA · 2018-10-10T02:41:03Z

Test build #97175 has finished for PR 21669 at commit e2f8063.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-10-10T06:45:07Z

Test build #97183 has finished for PR 21669 at commit a958920.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2018-10-10T16:45:32Z

...core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfExecutorFeatureStep.scala

+    HadoopBootstrapUtil.bootstrapHadoopConfDir(None, None, hadoopConfDirCMapName, pod)
+  }
+
+  override def getAdditionalPodSystemProperties(): Map[String, String] = Map.empty


No need to address it here but it feels like these methods should have default implementations, given that lots of classes just don't do anything with them.

vanzin · 2018-10-10T16:46:06Z

...src/main/scala/org/apache/spark/deploy/k8s/features/HadoopSparkUserExecutorFeatureStep.scala

+
+  override def configurePod(pod: SparkPod): SparkPod = {
+    val sparkUserName = kubernetesConf.sparkConf.get(KERBEROS_SPARK_USER_NAME)
+      HadoopBootstrapUtil.bootstrapSparkUserPod(sparkUserName, pod)


indentation

vanzin · 2018-10-10T16:46:23Z

...core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala

+  */
+private[spark] class KerberosConfDriverFeatureStep(
+    kubernetesConf: KubernetesConf[KubernetesDriverSpecificConf])
+    extends KubernetesFeatureConfigStep with Logging {


vanzin · 2018-10-10T16:56:07Z

docs/security.md

+
+In all cases you must define the environment variable: `HADOOP_CONF_DIR` or 
+`spark.kubernetes.hadoop.configMapName` as well as either
+`spark.kubernetes.kerberos.krb5.path` or `spark.kubernetes.kerberos.krb5.configMapName`.


This needs a small update after the recent changes.

vanzin · 2018-10-10T16:57:18Z

...core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala

+  }
+
+  override def getAdditionalKubernetesResources(): Seq[HasMetadata] = {
+    // HADOOP_CONF_DIR ConfigMap


Unnecessary comment.

vanzin · 2018-10-10T16:57:37Z

...core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala

+        fileLocation)
+    }
+
+    // Kerberos DT Secret


Unnecessary comment.

vanzin · 2018-10-10T16:59:08Z

...re/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopBootstrapUtil.scala

+
+private[spark] object HadoopBootstrapUtil extends Logging {
+
+   /**


indentation is wrong

vanzin · 2018-10-10T17:00:57Z

...re/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopBootstrapUtil.scala

+     SparkPod(pod.pod, envModifiedContainer)
+  }
+
+    /**


indentation is wrong

vanzin · 2018-10-10T17:01:11Z

...re/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopBootstrapUtil.scala

+    * @return a modified SparkPod
+    */
+  def bootstrapSparkUserPod(sparkUserName: String, pod: SparkPod): SparkPod = {
+     val envModifiedContainer = new ContainerBuilder(pod.container)


indentation is wrong

vanzin · 2018-10-10T17:02:36Z

...re/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopKerberosLogin.scala

+import org.apache.spark.deploy.k8s.Constants._
+import org.apache.spark.deploy.k8s.security.KubernetesHadoopDelegationTokenManager
+
+ /**


indentation is wrong.

Really, when I asked you to go through all your code, I meant all your code, not just the code I commented on. Please do that.

I apologize, these slip through the cracks. Definitely gonna add an additional linter / style-checker as the Jenkin's scalastyle checks seem to be quite primitive and does not complain. Thank you for patience in your review! I really appreciate it.

SparkQA · 2018-10-10T21:47:49Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3860/

SparkQA · 2018-10-10T21:57:22Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3860/

SparkQA · 2018-10-11T02:00:14Z

Test build #97220 has finished for PR 21669 at commit dd95fca.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2018-10-15T18:15:43Z

retest this please

vanzin · 2018-10-15T18:18:41Z

I went through the code again, and the main thing for me is that following this whole "steps" thing in the k8s backend still seems really hard to me. It feels like related code gets scattered around into a bunch of different places for no reason that I can detect.

e.g. adding the "HadoopConfigSpec" thing to "KubernetesConf" just seems to require a bunch of changes in unrelated places instead of just reading that information from the underlying configuration where it's actually needed.

But that's not going to be fixed here. When/if I have time I might try to figure this stuff out.

I still noticed some remaining style issues but at this point those don't matter. The functionality looks ok so, pending the new test run, I'll merge this.

SparkQA · 2018-10-15T18:27:57Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4000/

SparkQA · 2018-10-15T18:38:00Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4000/

SparkQA · 2018-10-15T22:44:45Z

Test build #97408 has finished for PR 21669 at commit dd95fca.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2018-10-15T22:48:31Z

Merging to master.

hustcat · 2019-02-15T08:07:45Z

Why closed? abandoned?

## What changes were proposed in this pull request? This is the work on setting up Secure HDFS interaction with Spark-on-K8S. The architecture is discussed in this community-wide google [doc](https://docs.google.com/document/d/1RBnXD9jMDjGonOdKJ2bA1lN4AAV_1RwpU_ewFuCNWKg) This initiative can be broken down into 4 Stages **STAGE 1** - [x] Detecting `HADOOP_CONF_DIR` environmental variable and using Config Maps to store all Hadoop config files locally, while also setting `HADOOP_CONF_DIR` locally in the driver / executors **STAGE 2** - [x] Grabbing `TGT` from `LTC` or using keytabs+principle and creating a `DT` that will be mounted as a secret or using a pre-populated secret **STAGE 3** - [x] Driver **STAGE 4** - [x] Executor ## How was this patch tested? Locally tested on a single-noded, pseudo-distributed Kerberized Hadoop Cluster - [x] E2E Integration tests apache#22608 - [ ] Unit tests ## Docs and Error Handling? - [x] Docs - [x] Error Handling ## Contribution Credit kimoonkim skonto Closes apache#21669 from ifilonenko/secure-hdfs. Lead-authored-by: Ilan Filonenko <if56@cornell.edu> Co-authored-by: Ilan Filonenko <ifilondz@gmail.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>

rvesse · 2019-02-19T18:27:19Z

@hustcat If you looked at the history above you would see that this was merged as part of
6c9c84f and is present in current master

The Closed status is an unfortunate artefact of how ASF infrastructure does mirroring between there and GitHub

This is the work on setting up Secure HDFS interaction with Spark-on-K8S. The architecture is discussed in this community-wide google [doc](https://docs.google.com/document/d/1RBnXD9jMDjGonOdKJ2bA1lN4AAV_1RwpU_ewFuCNWKg) This initiative can be broken down into 4 Stages **STAGE 1** - [x] Detecting `HADOOP_CONF_DIR` environmental variable and using Config Maps to store all Hadoop config files locally, while also setting `HADOOP_CONF_DIR` locally in the driver / executors **STAGE 2** - [x] Grabbing `TGT` from `LTC` or using keytabs+principle and creating a `DT` that will be mounted as a secret or using a pre-populated secret **STAGE 3** - [x] Driver **STAGE 4** - [x] Executor Locally tested on a single-noded, pseudo-distributed Kerberized Hadoop Cluster - [x] E2E Integration tests apache#22608 - [ ] Unit tests - [x] Docs - [x] Error Handling kimoonkim skonto Closes apache#21669 from ifilonenko/secure-hdfs. Lead-authored-by: Ilan Filonenko <if56@cornell.edu> Co-authored-by: Ilan Filonenko <ifilondz@gmail.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>

initial Driver logic for Hadoop and Kerberos Support

73f2777

add executors... still need to refactor to use sparkConf exclusivley

6069be5

skonto mentioned this pull request Jul 1, 2018

Add lightbend custom suite lightbend/spark#34

Merged

refactored executor logic preparing for e2e testing

000120f

skonto reviewed Jul 3, 2018

View reviewed changes

resolved initial comments

13b3adc

merge conflicts

0939738

hopefully final comment resolution

e2f8063

liyinan926 approved these changes Oct 9, 2018

View reviewed changes

vanzin reviewed Oct 9, 2018

View reviewed changes

ifilonenko added 2 commits October 9, 2018 17:36

style issues

f3a0ffb

included new ability to bake krb5.conf into your docker images and no…

a958920

…t fail

vanzin reviewed Oct 10, 2018

View reviewed changes

style check

dd95fca

asfgit closed this in 6c9c84f Oct 15, 2018


		private[spark] object HadoopBootstrapUtil extends Logging {

		/**

[SPARK-23257][K8S] Kerberos Support for Spark on K8S #21669

[SPARK-23257][K8S] Kerberos Support for Spark on K8S #21669

Uh oh!

Conversation

ifilonenko commented Jun 29, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Docs and Error Handling?

Contribution Credit

Uh oh!

SparkQA commented Jun 29, 2018

Uh oh!

SparkQA commented Jun 29, 2018

Uh oh!

SparkQA commented Jun 29, 2018

Uh oh!

felixcheung commented Jun 30, 2018

Uh oh!

SparkQA commented Jul 1, 2018

Uh oh!

SparkQA commented Jul 1, 2018

Uh oh!

SparkQA commented Jul 1, 2018

Uh oh!

SparkQA commented Jul 2, 2018

Uh oh!

SparkQA commented Jul 2, 2018

Uh oh!

SparkQA commented Jul 2, 2018

Uh oh!

skonto Jul 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skonto Jul 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skonto Jul 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skonto Jul 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skonto Jul 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 7, 2018

Uh oh!

SparkQA commented Jul 7, 2018

Uh oh!

SparkQA commented Jul 7, 2018

Uh oh!

SparkQA commented Jul 7, 2018

Uh oh!

SparkQA commented Jul 7, 2018

Uh oh!

SparkQA commented Jul 18, 2018

Uh oh!

SparkQA commented Jul 30, 2018

Uh oh!

SparkQA commented Jul 30, 2018

Uh oh!

SparkQA commented Jul 31, 2018

Uh oh!

SparkQA commented Oct 9, 2018

ifilonenko commented Jun 29, 2018 •

edited

Loading

skonto Jul 3, 2018 •

edited

Loading

skonto Jul 3, 2018 •

edited

Loading

skonto Jul 3, 2018 •

edited

Loading

skonto Jul 3, 2018 •

edited

Loading

skonto Jul 3, 2018 •

edited

Loading

vanzin Oct 9, 2018 •

edited

Loading