Migrating sayar/fabrikate-kafka-sample into a samples directory (#29)

* Initial commit

* Initial fab components for kafka and schema registry.

Added manifests for kafka cluster and schema registry user.
Added Kafka namespace yaml.
Added schema registry configuration using plaintext.
Added network policies.
Added azure-pipeline.yml for fab build and deploy.

* Fix typo in README.md.

* Using strimzi's network policies and cleaned up network policies.

* Updated fabrikate component type for kafka operator.

* Update README.md

Closes #2

* Added Prometheus and Grafana for Observability (#4)

* Added fabrikate component for prometheus and grafana
* Added metrics configuration
* added network policy for prometheus.
* Added sample dashboards for kafka and zookeeper for grafana.
* Added additional prometheus scrape config.
* Updated the README

* Use inline method. (#5)

* Setup kafka connect

* Initial support for portworx

* Add manifests for kafka-connect and kafka-connect-user

* Add sample mongoDB connector and a README file

* Refactored and renamed in prep for @cnadolny's PR. (#7)

* Refactored and renamed in prep for @cnadolny's PR.
* Moved client helpers to a different folder to distinguish them from local tests scripts.

* Change acl for the kafka connect user to a different topic

* Adding manual instructions around portworx setup

* Added mirror maker and documentation (#6)

* Enable prometheus metrics for kafka connect (#10)

* Commenting out portworx namespace and adding prometheus annotations

* Unix base

* Add setting for logs

* Strimzi Kafka Cluster deployment liveness test

* Refactoring portworx for easier navigation to deployment instructions

* Adding additional details for the Portworx Documentation

* DRYing livetest

* portworx snapshot and backup guidance init

* added zookeeper to allow full perf test testing

* Clarifying comments in test

* Working failover test, init snapshot test

* Adding Loadbalancer without TLS for external accessibilty

* External test running with kafkacat

* Renaming livetest to internaltest

* Refactored portworx, stable failover test, unstable backup test

* Update strimzi-px-install.sh

how to use env vars for secrets

* typo 💬

* adding test color output. Also using external bootstrap for test

* Modified tls ports for px-kafka brokers and docs correction

* Change config for logs

* First commit

* Readme updates

* Adding namespace

* Set up CI with Azure Pipelines

[skip ci]

* Set up CI with Azure Pipelines

[skip ci]

* updated tests and added timeoout in between each

* Add helm chart for component and config for kafka server logs

* Add helm chart files

* Add description to the fabrikate-kafka helm chart

* Updating image

* updating tests to use topic subdirectories. Also adding SSL support to external test

* Cleaned up and added optional flag for TLS for externaltest

* Adding optional config for tls enabled loadbalancer. Refactoring test README

* Adding support for portworx volume encryption

* replaced zookeeper with broker-list in perf tests

* Adding Kafka Connect Test (#21)

* Updating README to reflect mTLS support

* adding kafkaconnect test components

* fixing bug in internaltest

* removing extra logs from test

* Update README.md

* Update README.md

* Delete TODO.md

* Update README.md

* fixing broken link

* Revised portworx failover test

* Refactoring portworx-manual, updating docs for failover description

* removed kubectl apply for unneeded yaml file

* Updating portworx docs

* Update README.md

* Nesting fabrikate-kafka-sample code and documentation into samples/kafka-strimzi-operator

* renaming to kafka-strimzi-portworx

* adding newline at end of files

* Fixing portworx config values

.gitignore

@@ -1,3 +1,4 @@
+
 # Binaries for programs and plugins
 *.exe
 *.exe~

samples/kafka-strimzi-portworx/.gitignore

@@ -0,0 +1,5 @@
+test/temp
+generated/
+components/
+helm_repos/
+.DS_Store

samples/kafka-strimzi-portworx/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2019 Rami Sayar
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

samples/kafka-strimzi-portworx/README.md

@@ -0,0 +1,52 @@
+# fabrikate-kafka-sample
+
+This goal of this Fabrikate defintion is to provide an end to end example of running a secure stateful workload with backed by virtual storage on a single Kubernetes cluster.
+
+## Overview
+![](images/Kafka-Sample.png)
+
+The following components are deployed on to your AKS cluster:
+* Strimzi Kafka Operator (including Kafka Connect)
+* Confluent Schema Registry
+* Prometheus
+* Grafana
+* Stork & Portworx (Disabled by Default)
+
+
+## Installation Instructions
+1. Follow [instructions](/portworx-manual/README.md#setting-up-portworx-manually) to install Portworx manually on your Kubernetes cluster
+2. Use [Fabrikate](https://github.com/Microsoft/fabrikate) to install and generate this _high level definition_ to produce manifest yaml 
+3. Install [Grafana dashboards](#setting-up-grafana-dashboards) to build your "pane of glass" to enable observability.
+4. Run [tests](/test/README.md) to validate your installation 
+
+## Validations
+
+The repository provides multiple tests to help prove the end to end goal. We have tests that execute from within the cluster and from outside the cluster. Additionally load tests targeted at the virtual storage layer and end to end layer using TLS mutual authentication. 
+
+## Caveats
+
+Currently Portworx must be installed manually due to the lack of a Helm chart. Please refer to [instructions](#installation-instructions) on how to successfuly install this sample.
+
+## Future Work
+
+- [ ] Work with Portworx on Helm Chart
+- [ ] Use a secret store type target of Key Vault for Portworx Volume Secret
+- [ ] Confifgure TLS & SSL with PX-Secure for Portworx Pods
+- [ ] User authentication for accessing Portworx volumes
+- [ ] Get past possible VNET issues in Terraform template
+
+## Additional Notes
+
+This sample also sets up a default scalable Kafka Cluster configuration (see [manifests/kafka-cluster.yaml](./manifests/kafka-cluster.yaml)) with persistent volumes, three replicas and TLS mutual authentication.
+
+It also includes Kubernetes network policies to restrict traffic to the Kafka cluster (see [manifests/kafka-networkpolicy.yaml](./manifests/kafka-networkpolicies.yaml)). Only TLS traffic is permitted to the Kafka cluster. Plaintext is only permitted for the Confluent Schema Registry app.
+
+## Setting Up Grafana Dashboards
+
+The sample configuration provided does not expose grafana and prometheus metrics through an externally accessible IP. You may choose to create an external IP. Alternatively, you can connect to your cluster and port forward the grafana dashboard.
+
+`kubectl port-forward [POD NAME HERE grafana] -n grafana 3000`
+
+Browse to `localhost:3000/dashboard/import`.
+
+There are six dashboards [Kafka, Kafka Connect, Zookeeper Metrics, Portworx Node, Portworx Volume, and Portworx Cluster] included in the `dashboards/` folder. You can either copy paste the content of the json files or use the "Upload .json File".

samples/kafka-strimzi-portworx/azure-pipeline.yml

@@ -0,0 +1,36 @@
+trigger:
+- master
+
+pool:
+  vmImage: 'Ubuntu-16.04'
+
+steps:
+- checkout: self
+  persistCredentials: true
+  clean: true
+
+- bash: |
+    curl $BEDROCK_BUILD_SCRIPT > build.sh
+    chmod +x ./build.sh
+  displayName: Download Bedrock orchestration script
+  env:
+    BEDROCK_BUILD_SCRIPT: https://raw.githubusercontent.com/Microsoft/bedrock/master/gitops/azure-devops/build.sh
+
+- task: ShellScript@2
+  displayName: Validate fabrikate definitions
+  inputs:
+    scriptPath: build.sh
+  condition: eq(variables['Build.Reason'], 'PullRequest')
+  env:
+    VERIFY_ONLY: 1
+
+- task: ShellScript@2
+  displayName: Transform fabrikate definitions and publish to YAML manifests to repo
+  inputs:
+    scriptPath: build.sh
+  condition: ne(variables['Build.Reason'], 'PullRequest')
+  env:
+    ACCESS_TOKEN_SECRET: $(ACCESS_TOKEN)
+    COMMIT_MESSAGE: $(Build.SourceVersionMessage)
+    REPO: $(MANIFEST_REPO)
+    BRANCH_NAME: $(Build.SourceBranchName)

samples/kafka-strimzi-portworx/azure-pipelines.yml

@@ -0,0 +1,19 @@
+# Starter pipeline
+# Start with a minimal pipeline that you can customize to build and deploy your code.
+# Add steps that build, run tests, deploy, and more:
+# https://aka.ms/yaml
+
+trigger:
+- master
+
+pool:
+  vmImage: 'ubuntu-latest'
+
+steps:
+- script: echo Hello, world!
+  displayName: 'Run a one-line script'
+
+- script: |
+    echo Add other tasks to build, test, and deploy your project.
+    echo See https://aka.ms/yaml
+  displayName: 'Run a multi-line script'

samples/kafka-strimzi-portworx/component.yaml

@@ -0,0 +1,28 @@
+name: "kafka"
+generator: "static"
+path: "./manifests"
+subcomponents:
+  - name: "strimzi-kafka-operator"
+    type: "helm"
+    path: "strimzi-kafka-operator"
+    method: "helm"
+    source: "http://strimzi.io/charts/"
+  - name: "confluent-schema-registry"
+    type: "helm"
+    source: "https://kubernetes-charts-incubator.storage.googleapis.com/"
+    method: "helm"
+    path: "schema-registry"
+  - name: "prometheus-grafana"
+    source: "https://github.com/microsoft/fabrikate-definitions.git"
+    path: "definitions/fabrikate-prometheus-grafana"
+    method: "git"
+  - name: "kafka-cluster"
+    type: "helm"
+    source: ""
+    path: "/helm-chart/kafka-cluster"
+    method: "local"
+  # - name: "portworx"
+  #   type: helm
+  #   source: https://github.com/portworx/helm
+  #   method: git
+  #   path: charts/portworx