Update flink-demo to use operator dependencies

repository/flink/docs/demo/financial-fraud/README.md

@@ -12,7 +12,7 @@ The architecture follows more or less the [SMACK stack architecture](https://mes
 - Events: Event are being generated by a small [Golang generator](https://github.com/dcos/demos/blob/master/flink/1.11/generator/generator.go). The events are in the form 'Sunday, 23-Jul-17 01:06:47 UTC;66;26;7810', where the first field '23-Jul-17 01:06:47 UTC' represents the (increasing) timestamp of transactions; the second field '66' represent the sender account; the third field the receiver account; and the fourth field represent the dollar amount transferred during that transaction.
 - Ingestion: The generated events are being ingested and buffered by a Kafka queue with the default topic 'transactions'. Being a Microservice we will deploy the data-generator on kubernetes.
 - Stream Processing: As we require fast response times, we use Apache Flink as a Stream processor running the [FinancialTransactionJob](https://github.com/dcos/demos/tree/master/flink/1.10/flink-job/src/main/java/io/dcos).
-- Storage: Here we diverge a bit from the typical SMACK stack setup and don't write the results into a Datastore such as Apache Cassandra. Instead we write the results again into a Kafka Stream (default: 'fraud'). Note, that Kafka also offers data persistence for all unprocessed events.
+- Storage: Here we diverge a bit from the typical SMACK stack setup and don't write the results into a data store such as Apache Cassandra. Instead, we write the results again into a Kafka Stream (default: 'fraud'). Note, that Kafka also offers data persistence for all unprocessed events.
 - Actor: In order to view the results we use again a small [Golang viewer](https://github.com/dcos/demos/blob/master/flink/1.11/actor/actor_viewer.go) which simply reads and displays the results from the output Kafka stream. Being a Microservice we will deploy the viewer on kubernetes.
 
 ## Prerequisites
@@ -42,50 +42,25 @@ $ minikube start --vm-driver=hyperkit --cpus=6 --memory=9216 --disk-size=10g
 
 ### Install KUDO and the KUDO CLI
 
-- Have current KUDO CLI `v0.10.x` [installed](https://kudo.dev/docs/cli.html#setup-the-kudo-kubectl-plugin)
-    - Run `kubectl kudo version`, the output should look like:
+- Have current KUDO CLI `v0.15.x` [installed](https://kudo.dev/docs/cli.html#setup-the-kudo-kubectl-plugin)
+    - Run `kubectl kudo version`, the output should look something like:
         ```bash
         $ kubectl kudo version
-        KUDO Version: version.Info{GitVersion:"0.10.0", GitCommit:"d2310b12", BuildDate:"2020-01-10T18:54:59Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"darwin/amd64"}
+        KUDO Version: version.Info{GitVersion:"0.15.0", GitCommit:"3a6cdfa6", BuildDate:"2020-06-26T12:20:36Z", GoVersion:"go1.13", Compiler:"gc", Platform:"darwin/amd64"}
         ```
     - If not, upgrade to the latest version via `brew upgrade kudo-cli` on macOS or follow the [installation instructions](https://kudo.dev/docs/cli.html#setup-the-kudo-kubectl-plugin) on other platforms.
-    - Have current KUDO `v0.10.x` installed on your cluster:
+    - Have current KUDO `v0.15.x` installed on your cluster:
     - If you have KUDO already installed to the `kudo-system` namespace check its current Docker image tag to verify the version you are running:
-        - Make sure the output of `kubectl get pod kudo-controller-manager-0 -n kudo-system -o jsonpath='{.spec.containers[0].image}'` shows at least `v0.10.x` as Docker tag:
+        - Make sure the output of `kubectl get pod kudo-controller-manager-0 -n kudo-system -o jsonpath='{.spec.containers[0].image}'` shows at least `v0.15.x` as Docker tag:
         ```bash
         $ kubectl get pod kudo-controller-manager-0 -n kudo-system -o jsonpath='{.spec.containers[0].image}'
-        kudobuilder/controller:v0.10.0
+        kudobuilder/controller:v0.15.0
         ```
-    - If not, use the following commands to install KUDO `v0.10.x` to your cluster:
+    - If not, use the following commands to install KUDO `v0.15.x` to your cluster:
         ```bash
-        $ kubectl kudo init
+        $ kubectl kudo init --unsafe-self-signed-webhook-ca
         ```
-
-### Install Required Operators
-
-- Have the `zookeeper` Operator with `0.3.0` as OperatorVersion installed
-    Use the KUDO CLI with the following command:
-    ```bash
-    $ kubectl kudo install zookeeper --operator-version=0.3.0 --skip-instance
-    operator.kudo.dev/v1beta1/zookeeper created
-    operatorversion.kudo.dev/v1beta1/zookeeper-0.3.0 created
-    ```
-- Have the `kafka` Operator with `1.2.0` as OperatorVersion installed
-    Use the KUDO CLI with the following command:
-    ```bash
-    $ kubectl kudo install kafka --operator-version=1.2.0 --skip-instance
-    operator.kudo.dev/v1beta1/kafka created
-    operatorversion.kudo.dev/v1beta1/kafka-1.2.0 created
-    ```
-- Have the `flink` Operator with `0.2.1` as OperatorVersion installed
-    Use the KUDO CLI with the following command:
-    ```bash
-    $ kubectl kudo install flink --operator-version=0.2.1 --skip-instance
-    operator.kudo.dev/v1beta1/flink created
-    operatorversion.kudo.dev/v1beta1/flink-0.2.1 created
-    ```
-
-Now you should have all required Operators installed.
+      the `--unsafe-self-signed-webhook-ca` allows KUDO to run without the [cert-manager](https://github.com/jetstack/cert-manager) but rather with a self-signed CA bundle which is fine for the purposes of this demo.
 
 ## Start the Demo
 
@@ -102,11 +77,19 @@ Install the Flink demo objects straight out of the repository:
 
 ```bash
 $ kubectl kudo install repository/flink/docs/demo/financial-fraud/demo-operator --instance flink-demo
-operator.kudo.dev/v1beta1/flink-demo created
-operatorversion.kudo.dev/v1beta1/flink-demo-0.1.4 created
-instance.kudo.dev/v1beta1/flink-demo created
+operator default/flink-demo created
+operator default/kafka created
+operatorversion default/kafka-1.2.0 created
+operator default/zookeeper created
+operatorversion default/zookeeper-0.3.0 created
+operator default/flink created
+operatorversion default/flink-0.2.1 created
+operatorversion default/flink-demo-0.1.5 created
+instance default/flink-demo created
 ```
 
+The demo relies on [Zookeeper](https://github.com/kudobuilder/operators/tree/master/repository/zookeeper), [Kafka](https://github.com/kudobuilder/operators/tree/master/repository/kafka) and [Flink](https://github.com/kudobuilder/operators/tree/master/repository/flink) operators which are all automatically installed as part of the flink-demo operator.
+
 To see the status of the deploy plan for the Zookeeper operator we can utilize the CLI via:
 
 ```bash
@@ -257,8 +240,8 @@ To successfully uninstall the demo follow those steps:
 
 - Delete the `flink-demo` instance: `kubectl delete instance flink-demo`
 - Delete all PVCs:
-    - For Kafka: `kubectl delete pvc -l instance=kafka`
-    - For Zookeeper: `kubectl delete pvc -l instance=zk`
+    - For Kafka: `kubectl delete pvc -l kudo.dev/instance=kafka`
+    - For Zookeeper: `kubectl delete pvc -l kudo.dev/instance=zk`
 
 ## Command Reference
 
@@ -273,17 +256,11 @@ kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisione
 kubectl annotate storageclass --overwrite local-path storageclass.kubernetes.io/is-default-class=true
 
 # install KUDO manager
-kubectl kudo init --wait
+kubectl kudo init --unsafe-self-signed-webhook-ca --wait
 
 # verify correct version of manager
 kubectl get pod kudo-controller-manager-0 -n kudo-system -o jsonpath='{.spec.containers[0].image}'
 
-# install Operators and OperatorVersions
-kubectl kudo install zookeeper --operator-version=0.3.0 --skip-instance
-kubectl kudo install kafka --operator-version=1.2.0 --skip-instance
-kubectl kudo install flink --operator-version=0.2.1 --skip-instance
-
-
 # install demo
 kubectl kudo install repository/flink/docs/demo/financial-fraud/demo-operator --instance flink-demo
 

repository/flink/docs/demo/financial-fraud/demo-operator/operator.yaml

@@ -1,6 +1,6 @@
 name: "flink-demo"
-operatorVersion: "0.1.4"
-kudoVersion: 0.10.0
+operatorVersion: "0.1.5"
+kudoVersion: 0.10.0 # TODO: update to 0.15 once released
 kubernetesVersion: 1.15.0
 maintainers:
   - name: Fabian Baier
@@ -10,20 +10,26 @@ maintainers:
 url: https://zookeeper.apache.org/
 tasks:
   - name: kafka
-    kind: Apply
+    kind: KudoOperator
     spec:
-      resources:
-        - kafka.yaml
+      package: kafka
+      operatorVersion: 1.2.0
+      parameterFile: kafka-params.yaml
+      instanceName: kafka
   - name: zookeeper
-    kind: Apply
+    kind: KudoOperator
     spec:
-      resources:
-        - zookeeper.yaml
+      package: zookeeper
+      operatorVersion: 0.3.0
+      parameterFile: zookeeper-params.yaml
+      instanceName: zk
   - name: flink
-    kind: Apply
+    kind: KudoOperator
     spec:
-      resources:
-        - flink.yaml
+      package: flink
+      operatorVersion: 0.2.1
+      parameterFile: flink-params.yaml
+      instanceName: flink
   - name: generator
     kind: Apply
     spec:

repository/flink/docs/demo/financial-fraud/demo-operator/templates/flink-params.yaml

@@ -0,0 +1,3 @@
+high_availability: ZOOKEEPER
+zookeeper_url: "zk-zookeeper-0.zk-hs:2181,zk-zookeeper-1.zk-hs:2181,zk-zookeeper-2.zk-hs:2181"
+zookeeper_path: "/flink-demo-flink"

repository/flink/docs/demo/financial-fraud/demo-operator/templates/kafka-params.yaml

@@ -0,0 +1,5 @@
+ZOOKEEPER_URI: "zk-zookeeper-0.zk-hs:2181,zk-zookeeper-1.zk-hs:2181,zk-zookeeper-2.zk-hs:2181"
+ZOOKEEPER_PATH: "/flink-demo-kafka"
+BROKER_COUNT: "3"
+AUTO_CREATE_TOPICS_ENABLE: "true"
+BROKER_MEM: "1024Mi"

repository/flink/docs/demo/financial-fraud/demo-operator/templates/zookeeper-params.yaml

@@ -0,0 +1,2 @@
+CPUS: "0.3"
+MEMORY: "256Mi"