From 913c721df965bf0648ac97e45996c45be1a42561 Mon Sep 17 00:00:00 2001 From: anishakj <43978302+anishakj@users.noreply.github.com> Date: Wed, 24 Mar 2021 13:04:06 +0530 Subject: [PATCH] Issue 99: Updating documentation (#118) * Updating documentation Signed-off-by: anishakj * Fixed broken links Signed-off-by: anishakj * Added troubleshooting doc Signed-off-by: anishakj * Addressing review comments Signed-off-by: anishakj * Addressing review comments Signed-off-by: anishakj * Fixed broken links Signed-off-by: anishakj * fixed broken links Signed-off-by: anishakj * fixed broken links Signed-off-by: anishakj * addressing review comments Signed-off-by: anishakj * Fixed formatting Signed-off-by: anishakj --- CONTRIBUTING.md | 5 - README.md | 185 ++++----------------------- charts/bookkeeper-operator/README.md | 31 +++-- charts/bookkeeper/README.md | 114 +++++++++++++++-- doc/development.md | 15 ++- doc/troubleshooting.md | 82 ++++++++++++ 6 files changed, 236 insertions(+), 196 deletions(-) delete mode 100644 CONTRIBUTING.md create mode 100644 doc/troubleshooting.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md deleted file mode 100644 index eca502d5..00000000 --- a/CONTRIBUTING.md +++ /dev/null @@ -1,5 +0,0 @@ -# Contributing to Pravega Operator - - Please check the [Contributing](https://github.com/pravega/pravega-operator/wiki/Contributing) wiki page. - - Happy hacking! diff --git a/README.md b/README.md index 9e5b047b..10392c24 100644 --- a/README.md +++ b/README.md @@ -2,187 +2,48 @@ [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![GoDoc](https://godoc.org/github.com/pravega/bookkeeper-operator?status.svg)](https://godoc.org/github.com/pravega/bookkeeper-operator) [![Build Status](https://travis-ci.org/pravega/bookkeeper-operator.svg?branch=master)](https://travis-ci.org/pravega/bookkeeper-operator) [![Go Report](https://goreportcard.com/badge/github.com/pravega/bookkeeper-operator)](https://goreportcard.com/report/github.com/pravega/bookkeeper-operator) [![Version](https://img.shields.io/github/release/pravega/bookkeeper-operator.svg)](https://github.com/pravega/bookkeeper-operator/releases) -### Project status: alpha - -The project is currently alpha. While no breaking API changes are currently planned, we reserve the right to address bugs and change the API before the project is declared stable. - -## Table of Contents - - * [Overview](#overview) - * [Requirements](#requirements) - * [Quickstart](#quickstart) - * [Install the Operator](#install-the-operator) - * [Install the Operator in Test Mode](#install-the-operator-in-test-mode) - * [Install a sample Bookkeeper Cluster](#install-a-sample-bookkeeper-cluster) - * [Scale a Bookkeeper Cluster](#scale-a-bookkeeper-cluster) - * [Upgrade a Bookkeeper Cluster](#upgrade-a-bookkeeper-cluster) - * [Upgrade the Operator](#upgrade-the-operator) - * [Uninstall the Bookkeeper Cluster](#uninstall-the-bookkeeper-cluster) - * [Uninstall the Operator](#uninstall-the-operator) - * [Manual installation](#manual-installation) - * [Configuration](#configuration) - * [Development](#development) -* [Releases](#releases) - ## Overview [Bookkeeper](https://bookkeeper.apache.org/) is a scalable, fault-tolerant, and low-latency storage service optimized for real-time workloads. -The Bookkeeper Operator manages Bookkeeper clusters deployed to Kubernetes and automates tasks related to operating a Bookkeeper cluster. - -- [x] Create and destroy a Bookkeeper cluster -- [x] Resize cluster -- [x] Rolling upgrades - -## Requirements - -- Kubernetes 1.15+ -- Helm 3.2.1+ -- An existing Apache Zookeeper 3.6.1 cluster. This can be easily deployed using our [Zookeeper operator](https://github.com/pravega/zookeeper-operator) - -## Quickstart - -We recommend using our [helm charts](charts) for all installation and upgrades (but not for rollbacks at the moment since helm rollbacks are still experimental). The helm charts for bookkeeper operator (version 0.1.2 onwards) and bookkeeper cluster (version 0.5.0 onwards) are published in [https://charts.pravega.io](https://charts.pravega.io/). To add this repository to your Helm repos, use the following command: -``` -helm repo add pravega https://charts.pravega.io -``` -There are manual deployment, upgrade and rollback options available as well. - -### Install the Operator - -> Note: If you are running on Google Kubernetes Engine (GKE), please [check this first](doc/development.md#installation-on-google-kubernetes-engine). - -To understand how to deploy a Bookkeeper Operator using helm, refer to [this](charts/bookkeeper-operator#installing-the-chart). - -#### Install the Operator in Test Mode - The Operator can be run in `test mode` if we want to deploy the Bookkeeper Cluster on minikube or on a cluster with very limited resources by setting `testmode: true` in `values.yaml` file. Operator running in test mode skips the minimum replica requirement checks. Test mode provides a bare minimum setup and is not recommended to be used in production environments. - -### Install a sample Bookkeeper cluster +The Bookkeeper Operator manages Bookkeeper clusters deployed to Kubernetes and automates tasks related to operating a Bookkeeper cluster.The operator itself is built with the [Operator framework](https://github.com/operator-framework/operator-sdk). -To understand how to deploy a bookkeeper cluster using helm, refer to [this](charts/bookkeeper#installing-the-chart). +## Project status: alpha -Once the bookkeeper cluster with release name `bookkeeper` has been created, use the following command to verify that the cluster instances and its components are being created. - -``` -$ kubectl get bk -NAME VERSION DESIRED MEMBERS READY MEMBERS AGE -bookkeeper 0.7.0 3 1 25s -``` - -After a couple of minutes, all cluster members should become ready. - -``` -$ kubectl get bk -NAME VERSION DESIRED MEMBERS READY MEMBERS AGE -bookkeeper 0.7.0 3 3 2m -``` +The project is currently alpha. While no breaking API changes are currently planned, we reserve the right to address bugs and change the API before the project is declared stable. -``` -$ kubectl get all -l bookkeeper_cluster=bookkeeper -NAME READY STATUS RESTARTS AGE -pod/bookkeeper-bookie-0 1/1 Running 0 2m -pod/bookkeeper-bookie-1 1/1 Running 0 2m -pod/bookkeeper-bookie-2 1/1 Running 0 2m +## Install the Operator -NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE -service/bookkeeper-bookie-headless ClusterIP None 3181/TCP 2m +To understand how to deploy a Bookkeeper Operator refer to [Operator Deployment](charts/bookkeeper-operator#deploying-bookkeeper-operator). -NAME DESIRED CURRENT AGE -statefulset.apps/bookkeeper-bookie 3 3 2m -``` +## Upgrade the Operator -By default, a `BookkeeperCluster` is reachable using this kind of headless service URL for each pod: -``` -http://bookkeeper-bookie-0.bookkeeper-bookie-headless.bookkeeper-bookie:3181 -``` +For upgrading the bookkeeper operator check the document on [Operator Upgrade](doc/operator-upgrade.md) -### Scale a Bookkeeper cluster +## Features -You can scale Bookkeeper cluster by updating the `replicas` field in the BookkeeperCluster Spec. +- [x] [Create and destroy a Bookkeeper cluster](charts/bookkeeper/README.md) +- [x] [Resize cluster](charts/bookkeeper/README.md#updating-bookkeeper-cluster) +- [x] [Rolling upgrades/Rollback](doc/upgrade-cluster.md) +- [x] [Bookkeeper Configuration tuning](doc/configuration.md) +- [x] Input validation -Example of patching the Bookkeeper Cluster resource to scale the server instances to 4. +## Development -``` -kubectl patch bk bookkeeper --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": 4}]' -``` +Check out the [development guide](doc/development.md). -### Upgrade a Bookkeeper cluster +## Releases -Check out the [upgrade guide](doc/upgrade-cluster.md). +The latest Bookkeeper releases can be found on the [Github Release](https://github.com/pravega/bookkeeper-operator/releases) project page. -## Upgrade the Operator +## Contributing and Community -For upgrading the bookkeeper operator check the document [operator-upgrade](doc/operator-upgrade.md) - -### Uninstall the Bookkeeper cluster - -``` -$ helm uninstall [BOOKKEEPER_RELEASE_NAME] -``` - -Once the Bookkeeper cluster has been deleted, make sure to check that the zookeeper metadata has been cleaned up before proceeding with the deletion of the operator. This can be confirmed with the presence of the following log message in the operator logs. -``` -zookeeper metadata deleted -``` - -However, if the operator fails to delete this metadata from zookeeper, you will instead find the following log message in the operator logs. -``` -failed to cleanup [CLUSTER_NAME] metadata from zookeeper (znode path: /pravega/[PRAVEGA_CLUSTER_NAME]): -``` - -The operator additionally sends out a `ZKMETA_CLEANUP_ERROR` event to notify the user about this failure. The user can check this event by doing `kubectl get events`. The following is the sample describe output of the event that is generated by the operator in such a case -``` -Name: ZKMETA_CLEANUP_ERROR-nn6sd -Namespace: default -Labels: app=bookkeeper-cluster - bookkeeper_cluster=bookkeeper -Annotations: -API Version: v1 -Event Time: -First Timestamp: 2020-04-27T16:53:34Z -Involved Object: - API Version: app.k8s.io/v1beta1 - Kind: Application - Name: bookkeeper-cluster - Namespace: default -Kind: Event -Last Timestamp: 2020-04-27T16:53:34Z -Message: failed to cleanup bookkeeper metadata from zookeeper (znode path: /pravega/pravega): failed to delete zookeeper znodes for (bookkeeper): failed to connect to zookeeper: lookup zookeeper-client on 10.100.200.2:53: no such host -Metadata: - Creation Timestamp: 2020-04-27T16:53:34Z - Generate Name: ZKMETA_CLEANUP_ERROR- - Resource Version: 864342 - Self Link: /api/v1/namespaces/default/events/ZKMETA_CLEANUP_ERROR-nn6sd - UID: 5b4c3f80-36b5-43e6-b417-7992bc309218 -Reason: ZK Metadata Cleanup Failed -Reporting Component: bookkeeper-operator -Reporting Instance: bookkeeper-operator-6769886978-xsjx6 -Source: -Type: Error -Events: -``` - ->In case the operator fails to delete the zookeeper metadata, the user is expected to manually delete the metadata from zookeeper prior to reinstall. - -### Uninstall the Operator -> Note that the Bookkeeper clusters managed by the Bookkeeper operator will NOT be deleted even if the operator is uninstalled. - -``` -$ helm uninstall [BOOKKEEPER_OPERATOR_RELEASE_NAME] -``` - -### Manual installation - -You can also manually install/uninstall the operator and Bookkeeper with `kubectl` commands. Check out the [manual installation](doc/manual-installation.md) document for instructions. - -## Configuration - -Check out the [configuration document](doc/configuration.md). +We thrive to build a welcoming and open community for anyone who wants to use the operator or contribute to it. [Here](https://github.com/pravega/bookkeeper-operator/wiki/Contributing) we describe how to contribute to bookkeepe operator. Contact the developers and community on [slack](https://pravega-io.slack.com/) ([signup](https://pravega-slack-invite.herokuapp.com/)) if you need any help. -## Development +## Troubleshooting -Check out the [development guide](doc/development.md). +Check out the [bookkeeper troubleshooting](doc/troubleshooting.md#bookkeeper-cluster-issues) for bookkeeper issues and for operator issues [operator troubleshooting](doc/troubleshooting.md#bookkeeper-operator-issues). -## Releases +## License -The latest Bookkeeper releases can be found on the [Github Release](https://github.com/pravega/bookkeeper-operator/releases) project page. +Bookkeeper Operator is under Apache 2.0 license. See the [LICENSE](https://github.com/pravega/bookkeeper-operator/blob/master/LICENSE) for details. diff --git a/charts/bookkeeper-operator/README.md b/charts/bookkeeper-operator/README.md index 1633c65b..68e544ac 100644 --- a/charts/bookkeeper-operator/README.md +++ b/charts/bookkeeper-operator/README.md @@ -1,20 +1,16 @@ -# Bookkeeper Operator Helm Chart +# Deploying Bookkeeper-Operator -Installs [Bookkeeper Operator](https://github.com/pravega/bookkeeper-operator) to create/configure/manage Bookkeeper clusters atop Kubernetes. - -## Introduction - -This chart bootstraps a [Bookkeeper Operator](https://github.com/pravega/bookkeeper-operator) deployment on a [Kubernetes](http://kubernetes.io) cluster using the [Helm](https://helm.sh) package manager. The chart can be installed multiple times to create Bookkeeper Operator on multiple namespaces. +Here, we briefly describe how to install [Bookkeeper Operator](https://github.com/pravega/bookkeeper-operator) that is used to create/configure/manage Bookkeeper clusters atop Kubernetes. ## Prerequisites - Kubernetes 1.15+ with Beta APIs - Helm 3.2.1+ - - An existing Apache Zookeeper 3.6.1 cluster. This can be easily deployed using our [Zookeeper Operator](https://github.com/pravega/zookeeper-operator) - Cert-Manager v0.15.0+ or some other certificate management solution in order to manage the webhook service certificates. This can be easily deployed by referring to [this](https://cert-manager.io/docs/installation/kubernetes/) - An Issuer and a Certificate (either self-signed or CA signed) in the same namespace that the Bookkeeper Operator will be installed (refer to [this](https://github.com/pravega/bookkeeper-operator/blob/master/deploy/certificate.yaml) manifest to create a self-signed certificate in the default namespace) -## Installing the Chart +## Installing Bookkeeper-Operator +> Note: If you are running on Google Kubernetes Engine (GKE), please [check this first](../../doc/development.md#installation-on-google-kubernetes-engine). To install the bookkeeper-operator chart, use the following commands: ``` @@ -23,17 +19,24 @@ $ helm repo update $ helm install [RELEASE_NAME] pravega/bookkeeper-operator --version=[VERSION] --set webhookCert.certName=[CERT_NAME] --set webhookCert.secretName=[SECRET_NAME] ``` where: -- **[RELEASE_NAME]** is the release name for the bookkeeper-operator chart -- **[DEPLOYMENT_NAME]** is the name of the bookkeeper-operator deployment so created. (If [RELEASE_NAME] contains the string `bookkeeper-operator`, `[DEPLOYMENT_NAME] = [RELEASE_NAME]`, else `[DEPLOYMENT_NAME] = [RELEASE_NAME]-bookkeeper-operator`. The [DEPLOYMENT_NAME] can however be overridden by providing `--set fullnameOverride=[DEPLOYMENT_NAME]` along with the helm install command) -- **[VERSION]** can be any stable release version for bookkeeper-operator from 0.1.3 onwards + +- **[RELEASE_NAME]** is the release name for the bookkeeper-operator chart. +- **[VERSION]** can be any stable release version for bookkeeper-operator from 0.1.3 onwards. - **[CERT_NAME]** is the name of the certificate created as a prerequisite - **[SECRET_NAME]** is the name of the secret created by the above certificate -This command deploys a bookkeeper-operator on the Kubernetes cluster in its default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation. +This command deploys a pravega-operator on the Kubernetes cluster in its default configuration. The [configuration](#operator-configuration) section lists the parameters that can be configured during installation. + +>Note: If we provide [RELEASE_NAME] same as chart name, deployment name will be same as release-name. But if we are providing a different name for release(other than bookkeeper-operator in this case), deployment name will be [RELEASE_NAME]-[chart-name]. However, deployment name can be overridden by providing --set fullnameOverride=[DEPLOYMENT_NAME]` along with helm install command + >Note: If the bookkeeper-operator version is 0.1.2, webhookCert.certName and webhookCert.secretName should not be set. Also in this case, cert-manager and the certificate/issuer do not need to be deployed as prerequisites. -## Uninstalling the Chart +## Upgrading Bookkeeper-Operator + +For upgrading bookkeeper-operator, please refer [upgrade guide](../../doc/operator-upgrade.md) + +## Uninstalling Bookkeeper-Operator To uninstall/delete the bookkeeper-operator chart, use the following command: @@ -43,7 +46,7 @@ $ helm uninstall [RELEASE_NAME] This command removes all the Kubernetes components associated with the chart and deletes the release. -## Configuration +## Operator Configuration The following table lists the configurable parameters of the bookkeeper-operator chart and their default values. diff --git a/charts/bookkeeper/README.md b/charts/bookkeeper/README.md index 8fd8fe4b..355b20d9 100644 --- a/charts/bookkeeper/README.md +++ b/charts/bookkeeper/README.md @@ -1,21 +1,17 @@ -# Bookkeeper Helm Chart +# Deploying Bookkeeper -Installs Bookkeeper clusters atop Kubernetes. - -## Introduction - -This chart creates a Bookkeeper cluster in [Kubernetes](http://kubernetes.io) using the [Helm](https://helm.sh) package manager. The chart can be installed multiple times to create Bookkeeper cluster on multiple namespaces. +Here, we briefly describe how to [install](#installing-bookkeeper-cluster)/[update](#updating-bookkeeper-cluster)/[uninstall](#uninstalling-the-bookkeeper-cluster)/[configure](#configuration) bookkeeper clusters atop kubernetes. ## Prerequisites - Kubernetes 1.15+ with Beta APIs - Helm 3+ - An existing Apache Zookeeper 3.6.1 cluster. This can be easily deployed using our [Zookeeper Operator](https://github.com/pravega/zookeeper-operator) - - Bookkeeper Operator. You can install it using its own [Helm chart](https://github.com/pravega/bookkeeper-operator/tree/master/charts/bookkeeper-operator) + - Bookkeeper Operator. Please refer [this](../../charts/bookkeeper-operator/README.md) -## Installing the Chart +## Installing Bookkeeper Cluster -To install the bookkeeper chart, use the following commands: +To install the bookkeeper cluster, use the following commands: ``` $ helm repo add pravega https://charts.pravega.io @@ -24,12 +20,12 @@ $ helm install [RELEASE_NAME] pravega/bookkeeper --version=[VERSION] --set zooke ``` where: - **[RELEASE_NAME]** is the release name for the bookkeeper chart -- **[CLUSTER_NAME]** is the name of the bookkeeper cluster so created (if [RELEASE_NAME] contains the string `bookkeeper`, `[CLUSTER_NAME] = [RELEASE_NAME]`, else `[CLUSTER_NAME] = [RELEASE_NAME]-bookkeeper`. The [CLUSTER_NAME] can however be overridden by providing `--set fullnameOverride=[CLUSTER_NAME]` along with the helm install command) -- **[PRAVEGA_CLUSTER_NAME]** is the name of the pravega cluster (this field is optional and needs to be provided only if we expect the bookkeeper cluster to work with [Pravega](https://github.com/pravega/pravega) and if we wish to override its default value which is `pravega`) - **[VERSION]** can be any stable release version for bookkeeper from 0.5.0 onwards - **[ZOOKEEPER_HOST]** is the zookeeper service endpoint of your zookeeper cluster deployment (default value of this field is `zookeeper-client:2181`) - **[NAMESPACE]** is the namespace in which you wish to deploy the bookkeeper cluster (default value for this field is `default`) The bookkeeper cluster must be installed in the same namespace as the zookeeper cluster. +>Note: If we provide [RELEASE_NAME] same as chart name, cluster name will be same as release-name. But if we are providing a different name for release(other than bookkeeper in this case), cluster name will be [RELEASE_NAME]-[chart-name]. However, cluster name can be overridden by providing `--set fullnameOverride=[CLUSTER_NAME]` along with helm install command. + This command deploys bookkeeper on the Kubernetes cluster in its default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation. >Note: If the operator version is 0.1.3 or below and bookkeeper version is 0.9.0 or above, you would need to set the JVM options as shown below. @@ -37,9 +33,57 @@ This command deploys bookkeeper on the Kubernetes cluster in its default configu helm install [RELEASE_NAME] pravega/bookkeeper --version=[VERSION] --set zookeeperUri=[ZOOKEEPER_HOST] --set pravegaClusterName=[PRAVEGA_CLUSTER_NAME] -n [NAMESPACE] --set 'jvmOptions.extraOpts={-XX:+UseContainerSupport,-XX:+IgnoreUnrecognizedVMOptions}' ``` -## Uninstalling the Chart +Once the bookkeeper cluster with release name `bookkeeper` has been created, use the following command to verify that the cluster instances and its components are being created. + +``` +$ kubectl get bk +NAME VERSION DESIRED MEMBERS READY MEMBERS AGE +bookkeeper 0.7.0 3 1 25s +``` + +After a couple of minutes, all cluster members should become ready. + +``` +$ kubectl get bk +NAME VERSION DESIRED MEMBERS READY MEMBERS AGE +bookkeeper 0.7.0 3 3 2m +``` + +``` +$ kubectl get all -l bookkeeper_cluster=bookkeeper +NAME READY STATUS RESTARTS AGE +pod/bookkeeper-bookie-0 1/1 Running 0 2m +pod/bookkeeper-bookie-1 1/1 Running 0 2m +pod/bookkeeper-bookie-2 1/1 Running 0 2m + +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +service/bookkeeper-bookie-headless ClusterIP None 3181/TCP 2m + +NAME DESIRED CURRENT AGE +statefulset.apps/bookkeeper-bookie 3 3 2m +``` + +By default, a `BookkeeperCluster` is reachable using this kind of headless service URL for each pod: +``` +http://bookkeeper-bookie-0.bookkeeper-bookie-headless.bookkeeper-bookie:3181 +``` +## Updating Bookkeeper Cluster + +For updating the bookkeeper cluster, use the following command + +``` +helm upgrade [RELEASE_NAME] --version=[VERSION] --set replicas=5 +``` + we can also update other configurable parameters at run time. For changing options `minorCompactionInterval` to `1900` use the below command. -To uninstall/delete the bookkeeper chart, use the following command: + ``` + helm upgrade bookkeeper charts/bookkeeper --set-string options."minorCompactionInterval=1900" + ``` +Please refer [upgrade](../../doc/upgrade-cluster.md) for upgrading cluster versions. + +## Uninstalling the Bookkeeper cluster + +To uninstall/delete the bookkeeper cluster, use the following command: ``` $ helm uninstall [RELEASE_NAME] @@ -48,6 +92,50 @@ $ helm uninstall [RELEASE_NAME] This command removes all the Kubernetes components associated with the chart and deletes the release. > Note: If blockOwnerDeletion had been set to false during bookkeeper installation, the PVCs won't be removed automatically while uninstalling the bookkeeper chart, and would need to be deleted manually. +Once the Bookkeeper cluster has been deleted, make sure to check that the zookeeper metadata has been cleaned up before proceeding with the deletion of the operator. This can be confirmed with the presence of the following log message in the operator logs. +``` +zookeeper metadata deleted +``` + +However, if the operator fails to delete this metadata from zookeeper, you will instead find the following log message in the operator logs. +``` +failed to cleanup [CLUSTER_NAME] metadata from zookeeper (znode path: /pravega/[PRAVEGA_CLUSTER_NAME]): +``` + +The operator additionally sends out a `ZKMETA_CLEANUP_ERROR` event to notify the user about this failure. The user can check this event by doing `kubectl get events`. The following is the sample describe output of the event that is generated by the operator in such a case +``` +Name: ZKMETA_CLEANUP_ERROR-nn6sd +Namespace: default +Labels: app=bookkeeper-cluster + bookkeeper_cluster=bookkeeper +Annotations: +API Version: v1 +Event Time: +First Timestamp: 2020-04-27T16:53:34Z +Involved Object: + API Version: app.k8s.io/v1beta1 + Kind: Application + Name: bookkeeper-cluster + Namespace: default +Kind: Event +Last Timestamp: 2020-04-27T16:53:34Z +Message: failed to cleanup bookkeeper metadata from zookeeper (znode path: /pravega/pravega): failed to delete zookeeper znodes for (bookkeeper): failed to connect to zookeeper: lookup zookeeper-client on 10.100.200.2:53: no such host +Metadata: + Creation Timestamp: 2020-04-27T16:53:34Z + Generate Name: ZKMETA_CLEANUP_ERROR- + Resource Version: 864342 + Self Link: /api/v1/namespaces/default/events/ZKMETA_CLEANUP_ERROR-nn6sd + UID: 5b4c3f80-36b5-43e6-b417-7992bc309218 +Reason: ZK Metadata Cleanup Failed +Reporting Component: bookkeeper-operator +Reporting Instance: bookkeeper-operator-6769886978-xsjx6 +Source: +Type: Error +Events: +``` + +>In case the operator fails to delete the zookeeper metadata, the user is expected to manually delete the metadata from zookeeper prior to reinstall. + ## Configuration The following table lists the configurable parameters of the Bookkeeper chart and their default values. diff --git a/doc/development.md b/doc/development.md index 607d6c1b..f6288485 100644 --- a/doc/development.md +++ b/doc/development.md @@ -1,9 +1,17 @@ ## Development -Requirements: +## Contents + + * [Requirements](#requirements) + * [Build the operator image](#build-the-operator-image) + * [Run the Operator locally](#run-the-operator-locally) + * [Installation on Google Kubernetes Engine](#installation-on-google-kubernetes-engine) + * [Install the Operator in Test Mode](#install-the-operator-in-test-mode) + +### Requirements - Go 1.13+ -### Install Go +#### Install Go You can install go directly or use gvm ( go version manager) @@ -112,3 +120,6 @@ On GKE, the following command must be run before installing the Operator, replac ``` $ kubectl create clusterrolebinding your-user-cluster-admin-binding --clusterrole=cluster-admin --user=your.google.cloud.email@example.org ``` + +### Install the Operator in Test Mode + The Operator can be run in `test mode` if we want to deploy the Bookkeeper Cluster on minikube or on a cluster with very limited resources by setting `testmode: true` in `values.yaml` file. Operator running in test mode skips the minimum replica requirement checks. Test mode provides a bare minimum setup and is not recommended to be used in production environments. diff --git a/doc/troubleshooting.md b/doc/troubleshooting.md new file mode 100644 index 00000000..139f4a51 --- /dev/null +++ b/doc/troubleshooting.md @@ -0,0 +1,82 @@ +# Troubleshooting + +## Bookkeeper Cluster Issues + +* [Certificate Error: Internal error occurred: failed calling webhook](#certificate-error-internal-error-occurred-failed-calling-webhook) +* [Unsupported Bookkeeper cluster version](#unsupported-bookkeeper-cluster-version) +* [Unsupported upgrade from version](#unsupported-upgrade-from-version) +* [Invalid Cookie Exception](#invalid-cookie-exception) + +## Bookkeeper operator Issues +* [Operator pod in container creating state](#operator-pod-in-container-creating-state) + +## Certificate Error: Internal error occurred: failed calling webhook + +While installing bookkeeper, if we get the error as below, +``` +helm install bookkeeper charts/bookkeeper +Error: Post https://bookkeeper-webhook-svc.default.svc:443/validate-bookkeeper-pravega-io-v1alpha1-bookkeepercluster?timeout=30s: x509: certificate signed by unknown authority +``` +We need to ensure that certificates are installed before installing the operator. Please refer [prerequisite](../charts/bookkeeper-operator/README.md#Prerequisites) + +## Unsupported Bookkeeper cluster version + +While installing pravega, if we get the below error +``` +Error: admission webhook "bookkeeperwebhook.pravega.io" denied the request: unsupported Bookkeeper cluster version 0.10.0-2703.c9b7be114 +``` +We need to make sure the supported versions are present in config map by the following command + +`kubectl describe cm bk-supported-versions-map` + +If the entries are not there in configmap, we have to add these options in the configmap by enabling test mode as follows while installing operator + +``` +helm install bookkeeper-operator charts/bookkeeper-operator --set testmode.enabled=true --set testmode.version="0.10.0" +``` + +Alternatively, we can edit the configmap and add entry as `0.10.0:0.10.0` in the configmap and restart the bookkeeper-operator pod + +## Unsupported upgrade from version + +While upgrading bookkeeper, if we get the error similar to below + +``` +Error from server (unsupported upgrade from version 0.8.0-2640.e4c436ba9 to 0.9.0-2752.2652549b3): error when applying patch +``` +We need to make sure that supported versions are present in configmap as `0.8.0:0.9.0`. If the entries are missing, we have to add these options in the configmap by enabling test mode as follows while installing Operator + +If the version from which we are triggering upgrade is present in configmap, use the below command. +``` +helm install bookkeeper-operator charts/bookkeeper-operator --set testmode.enabled=true --set testmode.version="0.9.0" +``` +If the version from which we are triggering upgrade and the version to which upgrade is performed are not present in configmap use the below command. + +``` +helm install bookkeeper-operator charts/bookkeeper-operator --set testmode.enabled=true --set testmode.fromVersion="0.8.0" --set testmode.version="0.9.0" +``` +Alternatively, we can edit the configmap and add entry as `0.8.0:0.8.0,0.9.0` in the configmap and restart the bookkeeper-operator pod + +## Invalid Cookie Exception + +While installing bookkeeper, if the pods are not coming to ready state `1/1` and in the bookie logs if the error messages are seen as below, + +``` +2020-06-26 09:03:34,893 - ERROR - [main:Main@223] - Failed to build bookie server +org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: + at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:470) + at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:252) + at org.apache.bookkeeper.bookie.Bookie.(Bookie.java:691) + at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:137) + at org.apache.bookkeeper.proto.BookieServer.(BookieServer.java:106) + at org.apache.bookkeeper.server.service.BookieService.(BookieService.java:43) + at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:301) + at org.apache.bookkeeper.server.Main.doMain(Main.java:221) + at org.apache.bookkeeper.server.Main.main(Main.java:203) +``` + +we need to ensure that znode entries are cleaned up from previous installation. This can be done by either cleaning up znode entries from zookeeper nodes or by completely reinstalling zookeeper. + +## Operator pod in container creating state + +While installing operator, if the operator pod goes in `ContainerCreating` state for long time, make sure certificates are installed correctly. Please refer [prerequisite](../charts/bookkeeper-operator/README.md#Prerequisites)