Fix multiple KIC releases in helm

[Dean-Coakley]

Proposed changes

Fixes: #967
Fixes: #950

Allows multiple installs of the Ingress Controller to be installed in the same cluster.

Behaviour changes

• CustomResourceDefinitions are no longer managed by helm releases. Deleting a release will leave CRDs behind. - In line with helm chart best practices. - https://helm.sh/docs/chart_best_practices/custom_resource_definitions/
• CustomResourceDefinitions are installed unconditionally. Unless --skip-crds is provided to helm install.

Upgrades

Upgrade path with controller.enableCustomResources=true

• Given a deployed 1.7.x IC

• helm upgrade my-release deployments/helm-chart

  Note: Existing CRDs and CRs will be deleted. You will also see listers temporarily fail until CRDs are recreated.

• Recreate CRDs
  • kubectl apply -f deployments/helm-chart/crds
• Recreate CRs.

Upgrade path with controller.enableCustomResources=false

• Given a deployed 1.7.x IC

• helm upgrade my-release deployments/helm-chart

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto master
• I will ensure my PR is targeting the master branch and pulling from my branch from my own fork

[Rulox]

Looks good. I am a little bit worried by:
Note: Existing CRDs and CRs will be deleted. You will also see listers temporarily fail until CRDs are recreated.

I guess we'll have to document this very carefully on the new Release Notes.

[pleshakov]

@Dean-Coakley lgtm!

this doc https://github.com/nginxinc/kubernetes-ingress/blob/master/docs-web/installation/installation-with-helm.md needs to be updated as well before merging