Use a (VMware) Harbor hosted ha-proxy image for CAPD installations

[krishnade-vm]

Bug Report

We Found out that some DataCenter have rate limit by Docker , In such cases deployment of cluster fails . we need to Tackle this scenario/ Error from customer perspective

Error Log:
Unable to find image 'kindest/haproxy:2.1.1-alpine' locally
docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.
See 'docker run --help'.

Expected Behavior

Should be able to pull all necessary images that is required for cluster deployment

Steps to Reproduce the Bug

Deploy Ubuntu VM from DC (Please contact me if you need VM from our team DC )
deploy stand-alone cluster
you see progress stuck in control plane initialization

Used below debug method suggested by Josh

1. docker ps
   2.docker exec -it {bootstrap_hash} /bin/bash
   3.kubectl get po -A
   4.kubectl logs -f capd-controller-manager-xxxx -n capd-system -c manager

You will see the below error

Unable to find image 'kindest/haproxy:2.1.1-alpine' locally
docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.
See 'docker run --help'.

Environment Details

• Build version (tanzu version): 0.6.0rc
• Operating System (client):Ubuntu 20.04.2 LTS

[jpmcb]

We should pull all images from a VMware, public facing image proxy cache. We had talked about making this apart of the July release of TCE.

In the mean time, if you have access to a proxy cache, it can be used to get dockerhub images, cached within the registry, without hitting the rate limit. Then, you can pull them locally, and the docker agent should be able to find them and use them for your standalone cluster.

[joshrosso]

Adding on to @jpmcb's comment:

A fair near-term solution would be to:

a. ensure the haproxy image's repository is configurable
b. copy the image from DockerHub to Harbor, using a tool like imgpkg copy.

I'm also moving this to a feature request rather than bug. This is the expected
behavior of upstream CAPD.

[randomvariable]

See also kubernetes-sigs/kind#1895

[joshrosso]

Doing some initial investigation.

Making ha proxy image configurable:

• CAPD: make the kindest/haproxy image configurable kubernetes-sigs/cluster-api#4950

This capability is available in cluster-api v0.4.0.

Tanzu-framework is tracking the upgrade to cluster API v0.4.0 here:

• Upgrade Cluster API to 0.4.0 tanzu-framework#164

Once this is in place, we can update the CAPD provider in tanzu-framework to use an ha-proxy image pointed at our harbor registry.

We'll need to delay this request until after our 0.9.0 (public release).

The initial ask (although we should make this change to support customer throttling and air-gapped scenarios, came from our internal team. They were being rate-limited from within kind. Our recommended workaround for the near term is to pre-seed the host image with the ha-proxy image. This will prevent pulling it remotely.

cc @RusiH @krishnade-vm