Add support for ECR as the kubernetes repo

[aaroniscode]

Support already exists for a user to choose a private repository to use for the Kubernetes container images. For Cluster API AWS (CAPA), users may choose to leverage Amazon's hosted Docker container registry: Elastic Container Registry (ECR).

One use case for this is to use the VMware signed binaries. A user may may wish to push the VMware container images to ECR and create the CAPA AMI with the images pre-pulled.

kubeadm doesn't currently support authenticating with ECR: kubernetes/kubeadm#1820.

This PR makes the following changes:

1. Adds a packer variable iam_instance_profile that can assigned to the Packer builder. ECR is a private repository and requires IAM permissions assigned for access.
2. In the Ansible playbook, moves the providers role up earlier in the process. This is because the awscli is used to log into ECR and pull the images. Initially it was installed AFTER the kubernetes role. I read through the providers role and didn't see any issues with the change in ordering, but it would be good to have a second set of eyes confirm the ordering change is not an issue.
3. A check is added during the kubernetes role to determine in the kubernetes_container_registry is a Elastic Container Registry. If so, a new set of ansible tasks ecrpull.yml logs into ECR and obtains temporary credentials. Then each image is pulled using the temporary credentials.

/assign @detiber

[moshloop]

/hold

I think this use case is really specific, I think we need a mechanism to enable this, but it shouldn't be part of the core image-builder.

[aaroniscode]

@moshloop I don't think the use case is that specific. I'm making a few assumptions. Please let me know where you think my assumption is incorrect.

Assumption 1 -- Amazon Elastic Container Registry has widespread usage with AWS customers.

Assumption 2 -- Many (not an insignificant amount) users of image-builder will want to stamp images and NOT use the upstream container images hosted on Google Container Registry (GCR).

There may be a number of reasons why users would want to use their own or custom container images:

• Using a patch that is available to some Kubernetes component, but not yet in a formal Kubernetes release
• Using a vendor's signed images. Either for support or indemnity reasons

Any CAPA user that wants to use custom container images AND wants to use ECR would need the capability introduced in this PR.

I also think there is a valid use case to say this capability belongs in core image-builder:

1. Core image-builder has support for AWS. ECR is a core part of AWS just like EC2 which is fully supported in image-builder
2. According to one of the project Goals -- Support end users requirements to customize images for their business needs -- I think this falls into that goal.

Would be interested in your thoughts.

[moshloop]

I agree that the use case is valid, my concern is in the path to achieving the end goal.

If we add an option to disable pre-pulling the images then it opens up other approaches:

1. Consume a core image in another packer pipeline which includes the ecr pull step
2. Provide an option to supply an arbitrary ansible-playbook during the image build + patch mechanism for the packer configs.

If this is urgent and 1 is not suitable, then I am not opposed to merging this with a TODO deprecated comment to remove once the extension mechanism is available.

[frapposelli]

@moshloop this will eventually be fixed once containerd/containerd#6637 is closed.

ECR is a pretty popular container registry with users on AWS so I believe this PR should be treated as urgent if we want to foster utilization of image-builder for CAPA images

[frapposelli]

Adding comments.

/lgtm

[frapposelli]

Travis PTAL

/assign @codenrhoden

[codenrhoden]

/lgtm

I'd like to see the suggested comments incorporated so we don't lose the future TODO. I think this would also satisfy @moshloop request and according to #68 (comment), I'd like to see this one go in before the first tag. I know of end-users that will want this capability.

[frapposelli]

/assign @aaroniscode

PTAL

[aaroniscode]

thanks everyone for your feedback and review. I've incorporated the suggestions. I also rebased on master and tested using k8s 1.16.2 and private images on ECR.

[frapposelli]

/lgtm
/assign @moshloop

Moshe, does this look good to you?

[frapposelli]

ping @moshloop

[moshloop]

/unhold

[moshloop]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aaroniscode, moshloop

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• images/capi/OWNERS [moshloop]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment