Registry mirror with auth not working in Talos 1.8

[olavst-spk]

Bug Report

Description

I am using Talos in an air-gapped environment with Harbor as a caching proxy. The Harbor server is set up to require authentication and I provide the username / password in the Talos machine configuration.

This works fine in Talos 1.7, but in 1.8 it fails to pull the pause image and the cluster never becomes healthy.

I am using the following machine config:

machine:
  registries:
    config:
      harbor.example.com:
        tls:
          insecureSkipVerify: true # for testing
        auth:
          username: ${harbor_username}
          password: ${harbor_password}
    mirrors:
      gcr.io:
        overridePath: true
        endpoints:
          - https://harbor.example.com/v2/gcr
      ghcr.io:
        overridePath: true
        endpoints:
          - https://harbor.example.com/v2/ghcr
      registry.k8s.io:
        overridePath: true
        endpoints:
          - https://harbor.example.com/v2/k8s

Logs

$ talosctl logs kubelet
...
10.250.60.249: {"ts":1730814950534.9365,"caller":"internal/log.go:32","msg":"RunPodSandbox from runtime service failed","err":"rpc error: code = Unknown desc = failed to start sandbox \"1c66c7c4e23676ec50a326e927261714e2f7c55560d7543df94dad3b4e5f6132\": failed to get sandbox image \"registry.k8s.io/pause:3.10\": failed to pull image \"registry.k8s.io/pause:3.10\": failed to pull and unpack image \"registry.k8s.io/pause:3.10\": failed to resolve reference \"registry.k8s.io/pause:3.10\": unexpected status from HEAD request to https://harbor.example.com/v2/k8s/pause/manifests/3.10?ns=registry.k8s.io: 401 Unauthorized"}
10.250.60.249: {"ts":1730814950535.034,"caller":"kuberuntime/kuberuntime_sandbox.go:72","msg":"Failed to create sandbox for pod","pod":{"name":"kube-apiserver-plat02-control-plane-012ffb5d","namespace":"kube-system"},"err":"rpc error: code = Unknown desc = failed to start sandbox \"1c66c7c4e23676ec50a326e927261714e2f7c55560d7543df94dad3b4e5f6132\": failed to get sandbox image \"registry.k8s.io/pause:3.10\": failed to pull image \"registry.k8s.io/pause:3.10\": failed to pull and unpack image \"registry.k8s.io/pause:3.10\": failed to resolve reference \"registry.k8s.io/pause:3.10\": unexpected status from HEAD request to https://harbor.example.com/v2/k8s/pause/manifests/3.10?ns=registry.k8s.io: 401 Unauthorized"}
10.250.60.249: {"ts":1730814950535.0676,"caller":"kuberuntime/kuberuntime_manager.go:1170","msg":"CreatePodSandbox for pod failed","pod":{"name":"kube-apiserver-plat02-control-plane-012ffb5d","namespace":"kube-system"},"err":"rpc error: code = Unknown desc = failed to start sandbox \"1c66c7c4e23676ec50a326e927261714e2f7c55560d7543df94dad3b4e5f6132\": failed to get sandbox image \"registry.k8s.io/pause:3.10\": failed to pull image \"registry.k8s.io/pause:3.10\": failed to pull and unpack image \"registry.k8s.io/pause:3.10\": failed to resolve reference \"registry.k8s.io/pause:3.10\": unexpected status from HEAD request to https://harbor.example.com/v2/k8s/pause/manifests/3.10?ns=registry.k8s.io: 401 Unauthorized"}
10.250.60.249: {"ts":1730814950535.1875,"caller":"kubelet/pod_workers.go:1301","msg":"Error syncing pod, skipping","pod":{"name":"kube-apiserver-plat02-control-plane-012ffb5d","namespace":"kube-system"},"podUID":"1c67d13d087ea5ec05027c6c711d86ae","err":"failed to \"CreatePodSandbox\" for \"kube-apiserver-plat02-control-plane-012ffb5d_kube-system(1c67d13d087ea5ec05027c6c711d86ae)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"kube-apiserver-plat02-control-plane-012ffb5d_kube-system(1c67d13d087ea5ec05027c6c711d86ae)\\\": rpc error: code = Unknown desc = failed to start sandbox \\\"1c66c7c4e23676ec50a326e927261714e2f7c55560d7543df94dad3b4e5f6132\\\": failed to get sandbox image \\\"registry.k8s.io/pause:3.10\\\": failed to pull image \\\"registry.k8s.io/pause:3.10\\\": failed to pull and unpack image \\\"registry.k8s.io/pause:3.10\\\": failed to resolve reference \\\"registry.k8s.io/pause:3.10\\\": unexpected status from HEAD request to https://harbor.example.com/v2/k8s/pause/manifests/3.10?ns=registry.k8s.io: 401 Unauthorized\"","errCauses":[{"error":"failed to \"CreatePodSandbox\" for \"kube-apiserver-plat02-control-plane-012ffb5d_kube-system(1c67d13d087ea5ec05027c6c711d86ae)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"kube-apiserver-plat02-control-plane-012ffb5d_kube-system(1c67d13d087ea5ec05027c6c711d86ae)\\\": rpc error: code = Unknown desc = failed to start sandbox \\\"1c66c7c4e23676ec50a326e927261714e2f7c55560d7543df94dad3b4e5f6132\\\": failed to get sandbox image \\\"registry.k8s.io/pause:3.10\\\": failed to pull image \\\"registry.k8s.io/pause:3.10\\\": failed to pull and unpack image \\\"registry.k8s.io/pause:3.10\\\": failed to resolve reference \\\"registry.k8s.io/pause:3.10\\\": unexpected status from HEAD request to https://harbor.example.com/v2/k8s/pause/manifests/3.10?ns=registry.k8s.io: 401 Unauthorized\""}]}

Environment

• Talos version: 1.8.2
• Kubernetes version: 1.31.1
• Platform: vmware

[smira]

Duplicate of #9594

Workaround is to serve this image without auth (as it's a generic pause image).