Embedded registry mirror is broken on master

[brandond]

The spegel embedded registry mirror was broken by #10019, however this was not noticed as the embedded mirror e2e test is still disabled:

k3s/.github/workflows/e2e.yaml

Lines 39 to 40 in 307f07b

	# TODO fix embeddedmirror and add it to the matrix
	etest: [startup, s3, btrfs, externalip, privateregistry, wasm]

[brandond]

Doesn't need backports, regressing PR hasn't been backported yet. Will fix it in the backport.

[aganesh-suse]

Validated on master branch with version v1.30.2-rc2+k3s1

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"

$ uname -m
x86_64

Cluster Configuration:

HA: 3 server/ 1 agent

Config.yaml:

token: xxxx
cluster-init: true
write-kubeconfig-mode: "0644"
node-external-ip: 1.1.1.1
node-label:
- k3s-upgrade=server
embedded-registry: true

registry.yaml:

mirrors:
  private.registry.com:
    endpoint:
      - private.registry.com
  docker.io:
    endpoint:
      - private.registry.com      
  k8s.gcr.io:
    endpoint:
      - private.registry.com      
configs:
  private.registry.com:
    auth:
      username: <username>
      password: <password>
    tls:
      ca_file: /home/ubuntu/ca.pem

Testing Steps

1. Copy config.yaml and registries.yaml:

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s && sudo cp registries.yaml /etc/rancher/k3s

P.S: Remember to copy over the /home/ubuntu/ca.pem for the registry settings to work

2. Install k3s

curl -sfL https://get.k3s.io | sudo INSTALL_K3S_VERSION='v1.30.2-rc2+k3s1' sh -s - server

3. Verify Cluster Status:

kubectl get nodes -o wide
kubectl get pods -A

Validation Results:

• k3s version used for validation:

$ k3s -v
k3s version v1.30.2-rc2+k3s1 (b4d4ed8f)
go version go1.22.4

Sample spegel logs from the journal logs on the agent node:

$ journalctl -xeu k3s-agent | grep 'received image event'  
Jun 14 17:50:51 ip-172-31-24-105 k3s[208743]: time="2024-06-14T17:50:51Z" level=info msg="spegel 2024/06/14 17:50:51 \"level\"=0 \"msg\"=\"received image event\" \"image\"=\"docker.io/rancher/mirrored-pause:3.6@sha256:74bf6fc6be13c4ec53a86a5acf9fdbc6787b176db0693659ad6ac89f115e182c\" \"type\"=\"CREATE\""
Jun 14 17:50:51 ip-172-31-24-105 k3s[208743]: time="2024-06-14T17:50:51Z" level=info msg="spegel 2024/06/14 17:50:51 \"level\"=0 \"msg\"=\"received image event\" \"image\"=\"docker.io/rancher/mirrored-pause@sha256:74bf6fc6be13c4ec53a86a5acf9fdbc6787b176db0693659ad6ac89f115e182c\" \"type\"=\"CREATE\""
Jun 14 17:50:55 ip-172-31-24-105 k3s[208743]: time="2024-06-14T17:50:55Z" level=info msg="spegel 2024/06/14 17:50:55 \"level\"=0 \"msg\"=\"received image event\" \"image\"=\"private.registry.com/mytestcontainer:unprivileged@sha256:7e418465981575a9abef4ee16a80c562a2d2d171e591c1475c38347ef3ec2a72\" \"type\"=\"CREATE\""
Jun 14 17:50:55 ip-172-31-24-105 k3s[208743]: time="2024-06-14T17:50:55Z" level=info msg="spegel 2024/06/14 17:50:55 \"level\"=0 \"msg\"=\"received image event\" \"image\"=\"private.registry.com/mytestcontainer@sha256:7e418465981575a9abef4ee16a80c562a2d2d171e591c1475c38347ef3ec2a72\" \"type\"=\"CREATE\""
Jun 14 17:51:29 ip-172-31-24-105 k3s[208743]: time="2024-06-14T17:51:29Z" level=info msg="spegel 2024/06/14 17:51:29 \"level\"=0 \"msg\"=\"received image event\" \"image\"=\"private.registry.com/nginx:latest@sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498\" \"type\"=\"CREATE\""
Jun 14 17:51:29 ip-172-31-24-105 k3s[208743]: time="2024-06-14T17:51:29Z" level=info msg="spegel 2024/06/14 17:51:29 \"level\"=0 \"msg\"=\"received image event\" \"image\"=\"private.registry.com/nginx@sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498\" \"type\"=\"CREATE\""