How to use IAM role for registries.yaml mirrors

[Kellen275]

I have an rke2 1.28.11 cluster deployed in AWS.

I'd like to pull docker images from my docker registry, but fallback to ECR if the image cannot be found. I have the following registries.yaml

mirros:
  my.registry:
    endpoint:
      - 123456789.my.ecr.endpoint

With this, I can see the fallback behavior working in containerd logs

... error="failed to pull and unpack image \"my.registry\" ...
... msg="trying next host" error="pull access denied ... authorization failed: no basic auth credentials" host=123456789.my.ecr.endpoint

The cluster is able to pull directly from ECR as a result of having the cloud-provider flags. My understanding is that this leverages the attached IAM role.

Is there a way to specify the same authentication method in registries.yaml as what's otherwise used when trying to pull directly from ECR?

[brandond]

The cluster is able to pull directly from ECR as a result of having the cloud-provider flags.

No, it is not. There is no longer any cloud-provider-specific code embedded in Kubernetes. There has not been for quite a while. Neither containerd nor the kubelet know how to use IAM roles or anything else like that to obtain credentials. You should deploy a Kubelet image credential provider plugin if you want to use IAM roles instead of static credentials.

• https://kubernetes.io/docs/tasks/administer-cluster/kubelet-credential-provider/
• https://github.com/kubernetes/cloud-provider-aws/tree/master/cmd/ecr-credential-provider
• https://docs.k3s.io/cli/agent

--image-credential-provider-bin-dir value  (agent/node) The path to the directory where credential provider plugin binaries are located (default: "/var/lib/rancher/credentialprovider/bin")
--image-credential-provider-config value   (agent/node) The path to the credential provider plugin config file (default: "/var/lib/rancher/credentialprovider/config.yaml")

[Kellen275]

Apologies. Yes, I can confirm I have the kubelet credential provider installed and configured according to the default values you listed.

Should I not then expect this config to be used during the retry after failing to pull from my first host?

Ultimately I'm surprised that a pod with image: 123456789.my.ecr.endpoint/busybox:latest succeeds whereas image: my.registry/busybox:latest fails given the image doesn't exist in my.registry but I have the registries.yaml from my initial post

[brandond]

You are dealing with different layers that are not aware of each other.

Containerd is not aware of the kubelet credential provider. The kubelet invokes the credential provider to obtain credentials from a registry that matches a pattern configured in the credential provider config file, and passes them along to containerd when pulling the image. As far as containerd is concerned, this is the same as if you were using ImagePullSecrets in the pod spec, or passed credentials on the command line when doing crictl image pull.

If you try to use ECR as a mirror for a different registry in the containerd config, the kubelet is not aware of that. It will not obtain credentials from the plugin and pass them to containerd, because as far as it knows you are not pulling from ECR.

[Kellen275]

Understood, I appreciate the additional clarification 🙂