changes for controller runtime upgrade (#72)

* Initial changes for controller runtime upgrade Signed-off-by: anisha.kj <anisha.kj@dell.com> * Made documentation and chart changes Signed-off-by: anisha.kj <anisha.kj@dell.com> * Added upgrade changes for operator Signed-off-by: anisha.kj <anisha.kj@dell.com> * Fixed typo Signed-off-by: anisha.kj <anisha.kj@dell.com> * modified the doc Signed-off-by: anisha.kj <anisha.kj@dell.com> * Added steps for manual upgrade Signed-off-by: anisha.kj <anisha.kj@dell.com> * Addressed review comments Signed-off-by: anisha.kj <anisha.kj@dell.com> * Changed the name of certificate Signed-off-by: anisha.kj <anisha.kj@dell.com>
pravega · Sep 14, 2020 · 8af58c5 · 8af58c5
1 parent 659f239
commit 8af58c5
Show file tree

Hide file tree

Showing 33 changed files with 2,362 additions and 818 deletions.
diff --git a/README.md b/README.md
@@ -37,7 +37,7 @@ The Bookkeeper Operator manages Bookkeeper clusters deployed to Kubernetes and a
 ## Requirements
 
 - Kubernetes 1.15+
-- Helm 3+
+- Helm 3.2.1+
 - An existing Apache Zookeeper 3.6.1 cluster. This can be easily deployed using our [Zookeeper operator](https://github.com/pravega/zookeeper-operator)
 
 ## Quickstart
@@ -46,20 +46,7 @@ The Bookkeeper Operator manages Bookkeeper clusters deployed to Kubernetes and a
 
 > Note: If you are running on Google Kubernetes Engine (GKE), please [check this first](doc/development.md#installation-on-google-kubernetes-engine).
 
-Use Helm to quickly deploy a Bookkeeper operator with the release name `bookkeeper-operator`.
-
-```
-$ helm install bookkeeper-operator charts/bookkeeper-operator
-```
-
-Verify that the Bookkeeper Operator is running.
-
-```
-$ kubectl get deploy
-NAME                          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE     AGE
-bookkeeper-operator              1         1         1            1          17s
-```
-
+We recommend using helm to deploy a Bookkeeper Operator. Check out the [helm installation](charts/bookkeeper-operator/README.md) document for instructions.
 #### Install the Operator in Test Mode
  The Operator can be run in `test mode` if we want to deploy the Bookkeeper Cluster on minikube or on a cluster with very limited resources by setting `testmode: true` in `values.yaml` file. Operator running in test mode skips the minimum replica requirement checks. Test mode provides a bare minimum setup and is not recommended to be used in production environments.
 
@@ -139,14 +126,7 @@ Check out the [upgrade guide](doc/upgrade-cluster.md).
 
 ## Upgrade the Operator
 
-Bookkeeper operator can be upgraded via helm using the following command
-```
-$ helm upgrade bookkeeper-operator <location of modified operator charts>
-```
-Here `bookkeeper-operator` is the release name of the operator. It can also be upgraded manually by modifying the image tag using the following command
-```
-$ kubectl edit deploy bookkeeper-operator
-```
+For upgrading the bookkeeper operator check the document [operator-upgrade](doc/operator-upgrade.md)
 
 ### Uninstall the Bookkeeper cluster
 

diff --git a/charts/bookkeeper-operator/README.md b/charts/bookkeeper-operator/README.md
@@ -8,15 +8,23 @@ This chart bootstraps a [Bookkeeper Operator](https://github.com/pravega/bookkee
 
 ## Prerequisites
   - Kubernetes 1.15+ with Beta APIs
-  - Helm 3+
+  - Helm 3.2.1+
   - An existing Apache Zookeeper 3.6.1 cluster. This can be easily deployed using our [Zookeeper Operator](https://github.com/pravega/zookeeper-operator)
+  - Cert-Manager v0.15.0+ or some other certificate management solution in order to manage the webhook service certificates. This can be easily deployed by referring to [this](https://cert-manager.io/docs/installation/kubernetes/)
+- An Issuer and a Certificate (either self-signed or CA signed) in the same namespace that the Bookkeeper Operator will be installed (refer to [this](https://github.com/pravega/bookkeeper-operator/blob/master/deploy/certificate.yaml) manifest to create a self-signed certificate in the default namespace)
+> The name of the certificate (*webhookCert.certName*), the name of the secret created by this certificate (*webhookCert.secretName*), the tls.crt (*webhookCert.crt*) and tls.key (*webhookCert.key*) need to be specified against the corresponding fields in the values.yaml file, or can be provided with the install command as shown [here](#installing-the-chart).
+The values *tls.crt* and *tls.key* are contained in the secret which is created by the certificate and can be obtained using the following command
+```
+kubectl get secret <secret-name> -o yaml | grep tls.
+```
+
 
 ## Installing the Chart
 
 To install the chart with the release name `my-release`:
 
 ```
-$ helm install my-release bookkeeper-operator
+$ helm install my-release bookkeeper-operator --set webhookCert.generate=false --set webhookCert.certName=<cert-name> --set webhookCert.secretName=<secret-name>
 ```
 
 The command deploys bookkeeper operator on the Kubernetes cluster in the default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation.
@@ -46,4 +54,9 @@ The following table lists the configurable parameters of the Bookkeeper operator
 | `serviceAccount.name` | Name for the service account | `bookkeeper-operator` |
 | `testmode.enabled` | Enable test mode | `false` |
 | `testmode.version` | Major version number of the alternate bookkeeper image we want the operator to deploy, if test mode is enabled | `""` |
+| `webhookCert.crt` | tls.crt value corresponding to the certificate | |
+| `webhookCert.key` | tls.key value corresponding to the certificate | |
+| `webhookCert.generate` | Whether to generate the certificate and the issuer (set to false while using self-signed certificates) | `false` |
+| `webhookCert.certName` | Name of the certificate, if generate is set to false | `selfsigned-cert-bk` |
+| `webhookCert.secretName` | Name of the secret created by the certificate, if generate is set to false | `selfsigned-cert-tls-bk` |
 | `watchNamespace` | Namespaces to be watched  | `""` |
diff --git a/charts/bookkeeper-operator/templates/certificate.yaml b/charts/bookkeeper-operator/templates/certificate.yaml
@@ -0,0 +1,46 @@
+{{- if .Values.webhookCert.generate }}
+apiVersion: cert-manager.io/v1alpha2
+kind: Issuer
+metadata:
+  name: {{ template "bookkeeper-operator.fullname" . }}-issuer
+  namespace: {{ .Release.Namespace }}
+  labels:
+{{ include "bookkeeper-operator.commonLabels" . | indent 4 }}
+spec:
+  ca:
+    secretName: {{ template "bookkeeper-operator.fullname" . }}-key-pair
+{{- end }}
+---
+{{- if .Values.webhookCert.generate }}
+apiVersion: cert-manager.io/v1alpha2
+kind: Certificate
+metadata:
+  name: {{ template "bookkeeper-operator.fullname" . }}-cert-bk
+  namespace: {{ .Release.Namespace }}
+  labels:
+{{ include "bookkeeper-operator.commonLabels" . | indent 4 }}
+spec:
+  isCA: true
+  duration: 8760h
+  renewBefore: 360h
+  secretName: {{ template "bookkeeper-operator.fullname" . }}-cert-tls-bk
+  commonName: bookkeeper-webhook-svc.{{ .Release.Namespace }}.svc.cluster.local
+  dnsNames:
+    - bookkeeper-webhook-svc
+    - bookkeeper-webhook-svc.{{ .Release.Namespace }}.svc.cluster.local
+    - bookkeeper-webhook-svc.{{ .Release.Namespace }}.svc
+  issuerRef:
+    name: {{ template "bookkeeper-operator.fullname" . }}-issuer
+{{- end }}
+---
+{{- if .Values.webhookCert.generate }}
+apiVersion: v1
+kind: Secret
+metadata:
+  name: {{ template "bookkeeper-operator.fullname" . }}-key-pair
+  namespace: {{ .Release.Namespace }}
+type: kubernetes.io/tls
+data:
+  tls.crt: {{ .Values.webhookCert.crt }}
+  tls.key: {{ .Values.webhookCert.key }}
+{{- end }}
diff --git a/charts/bookkeeper-operator/templates/operator.yaml b/charts/bookkeeper-operator/templates/operator.yaml
@@ -39,9 +39,19 @@ spec:
         - name: OPERATOR_NAME
           value: {{ template "bookkeeper-operator.fullname" . }}
         volumeMounts:
+        - name: webhook-cert
+          mountPath: "/tmp/k8s-webhook-server/serving-certs"
+          readOnly: true
         - name: versions-volume
           mountPath: "/tmp/config"
       volumes:
+      - name: webhook-cert
+        secret:
+          {{- if .Values.webhookCert.generate }}
+          secretName: {{ template "bookkeeper-operator.fullname" . }}-cert-tls-bk
+          {{- else }}
+          secretName: {{ .Values.webhookCert.secretName }}
+          {{- end }}
       - name: versions-volume
         configMap:
           name: bk-supported-versions-map
diff --git a/charts/bookkeeper-operator/templates/webhook.yaml b/charts/bookkeeper-operator/templates/webhook.yaml
@@ -0,0 +1,45 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: bookkeeper-webhook-svc
+  namespace: {{ .Release.Namespace }}
+  labels:
+{{ include "bookkeeper-operator.commonLabels" . | indent 4 }}
+spec:
+  ports:
+  - port: 443
+    protocol: TCP
+    targetPort: 443
+  selector:
+    component: bookkeeper-operator
+  sessionAffinity: None
+  type: ClusterIP
+---
+
+apiVersion: admissionregistration.k8s.io/v1beta1
+kind: ValidatingWebhookConfiguration
+metadata:
+  name: bookkeeper-webhook-config
+  labels:
+{{ include "bookkeeper-operator.commonLabels" . | indent 4 }}
+  annotations:
+    cert-manager.io/inject-ca-from: {{ .Release.Namespace }}/{{ .Values.certName }}
+webhooks:
+- clientConfig:
+    service:
+      name: bookkeeper-webhook-svc
+      namespace: {{ .Release.Namespace }}
+      path: /validate-bookkeeper-pravega-io-v1beta1-bookkeepercluster
+  name: bookkeeperwebhook.pravega.io
+  rules:
+  - apiGroups:
+    - bookkeeper.pravega.io
+    apiVersions:
+    - v1alpha1
+    operations:
+    - CREATE
+    - UPDATE
+    resources:
+    - bookkeeperclusters
+    scope: "*"
+  timeoutSeconds: 30
diff --git a/charts/bookkeeper-operator/values.yaml b/charts/bookkeeper-operator/values.yaml
@@ -4,7 +4,7 @@
 
 image:
   repository: pravega/bookkeeper-operator
-  tag: 0.1.2
+  tag: 0.1.2 
   pullPolicy: IfNotPresent
 
 ## Install RBAC roles and bindings.
@@ -27,6 +27,15 @@ testmode:
   ## eg. enter 0.8.0 if u wish to deploy version 0.8.0-2500.efe501a
   version: ""
 
+webhookCert:
+  crt:
+  key:
+  ## Whether to generate the certificate or not.
+  generate: false
+  ## If false, the name of the pre-existing certificate needs to be provided.
+  certName: selfsigned-cert-bk
+  secretName: selfsigned-cert-tls-bk
+
 ## Specifies which namespace the Operator should watch over.
 ## An empty string means all namespaces.
 watchNamespace: ""

diff --git a/cmd/manager/main.go b/cmd/manager/main.go
@@ -18,13 +18,14 @@ import (
 
 	"github.com/operator-framework/operator-sdk/pkg/k8sutil"
 	"github.com/operator-framework/operator-sdk/pkg/leader"
-	"github.com/operator-framework/operator-sdk/pkg/ready"
 	sdkVersion "github.com/operator-framework/operator-sdk/version"
+
 	"github.com/pravega/bookkeeper-operator/pkg/apis"
+	"github.com/pravega/bookkeeper-operator/pkg/apis/bookkeeper/v1alpha1"
+
 	"github.com/pravega/bookkeeper-operator/pkg/controller"
 	controllerconfig "github.com/pravega/bookkeeper-operator/pkg/controller/config"
 	"github.com/pravega/bookkeeper-operator/pkg/version"
-	"github.com/pravega/bookkeeper-operator/pkg/webhook"
 	log "github.com/sirupsen/logrus"
 	_ "k8s.io/client-go/plugin/pkg/client/auth/gcp"
 	_ "k8s.io/client-go/plugin/pkg/client/auth/oidc"
@@ -79,13 +80,6 @@ func main() {
 	// Become the leader before proceeding
 	leader.Become(context.TODO(), "bookkeeper-operator-lock")
 
-	r := ready.NewFileReady()
-	err = r.Set()
-	if err != nil {
-		log.Fatal(err, "")
-	}
-	defer r.Unset()
-
 	// Create a new Cmd to provide shared dependencies and start components
 	mgr, err := manager.New(cfg, manager.Options{Namespace: namespace})
 	if err != nil {
@@ -108,11 +102,11 @@ func main() {
 
 	if webhookFlag {
 		// Setup webhook
-		if err := webhook.AddToManager(mgr); err != nil {
-			log.Fatal(err)
+		if err := (&v1alpha1.BookkeeperCluster{}).SetupWebhookWithManager(mgr); err != nil {
+			log.Error(err, "unable to create webhook %s", err.Error())
+			os.Exit(1)
 		}
 	}
-
 	log.Print("Webhook Setup completed")
 	log.Print("Starting the Cmd")
 

diff --git a/deploy/certificate.yaml b/deploy/certificate.yaml
@@ -0,0 +1,22 @@
+apiVersion: cert-manager.io/v1alpha2
+kind: Issuer
+metadata:
+  name: selfsigned-issuer-bk
+  namespace: default
+spec:
+  selfSigned: {}
+---
+apiVersion: cert-manager.io/v1alpha2
+kind: Certificate
+metadata:
+  name: selfsigned-cert-bk
+  namespace: default
+spec:
+  secretName: selfsigned-cert-tls-bk
+  commonName: bookkeeper-webhook-svc.default.svc.cluster.local
+  dnsNames:
+    - bookkeeper-webhook-svc 
+    - bookkeeper-webhook-svc.default.svc.cluster.local
+    - bookkeeper-webhook-svc.default.svc
+  issuerRef:
+    name: selfsigned-issuer-bk
diff --git a/deploy/operator.yaml b/deploy/operator.yaml
@@ -16,7 +16,7 @@ spec:
       serviceAccountName: bookkeeper-operator
       containers:
         - name: bookkeeper-operator
-          image: pravega/bookkeeper-operator:latest
+          image: pravega/bookkeeper-operator:0.1.2
           ports:
           - containerPort: 60000
             name: metrics
@@ -33,9 +33,15 @@ spec:
             - name: OPERATOR_NAME
               value: "bookkeeper-operator"
           volumeMounts:
+            - name: webhook-cert
+              mountPath: "/tmp/k8s-webhook-server/serving-certs"
+              readOnly: true
             - name: versions-volume
               mountPath: "/tmp/config"
       volumes:
+        - name: webhook-cert
+          secret:
+            secretName: selfsigned-cert-tls-bk
         - name: versions-volume
           configMap:
             name: bk-supported-versions-map
diff --git a/deploy/webhook.yaml b/deploy/webhook.yaml
@@ -0,0 +1,41 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: bookkeeper-webhook-svc
+  namespace: default
+spec:
+  ports:
+  - port: 443
+    protocol: TCP
+    targetPort: 443
+  selector:
+    component: bookkeeper-operator
+  sessionAffinity: None
+  type: ClusterIP
+---
+
+apiVersion: admissionregistration.k8s.io/v1beta1
+kind: ValidatingWebhookConfiguration
+metadata:
+  name: bookkeeper-webhook-config
+  annotations:
+    cert-manager.io/inject-ca-from: default/selfsigned-cert-bk
+webhooks:
+- clientConfig:
+    service:
+      name: bookkeeper-webhook-svc
+      namespace: default
+      path: /validate-bookkeeper-pravega-io-v1alpha1-bookkeepercluster
+  name: bookkeeperwebhook.pravega.io
+  rules:
+  - apiGroups:
+    - bookkeeper.pravega.io
+    apiVersions:
+    - v1alpha1
+    operations:
+    - CREATE
+    - UPDATE
+    resources:
+    - bookkeeperclusters
+    scope: "*"
+  timeoutSeconds: 30
diff --git a/doc/manual-installation.md b/doc/manual-installation.md
@@ -10,6 +10,19 @@
 
 > Note: If you are running on Google Kubernetes Engine (GKE), please [check this first](#installation-on-google-kubernetes-engine).
 
+In case you dont have a cert-manager, Install the cert-manager v0.15.0+ from the following link:-
+
+https://cert-manager.io/docs/installation/kubernetes/
+
+Install the issuer and certificate
+```
+$ kubectl create -f deploy/certificate.yaml
+```
+Install the webhook
+```
+$ kubectl create -f deploy/webhook.yaml  
+```
+
 Register the Bookkeeper cluster custom resource definition (CRD).
 
 ```