K3D Master + K3S Worker Nodes - Is this possible?

[darktempla]

Hi all,

Thanks in advance for any responses to my discussion.

What am I trying to do?

Spin up a k3d single node cluster (master only) and connect external k3s worker nodes to this cluster.

Why am I trying to do this?

I have found having a raspberry pi as a K3S master over time the CPU gets overworked and becomes un-realiable. So I am attempting to get some stability into my cluster without breaking the bank and use what I currently have.

I have an M1 Mac Mini that has plenty of grunt so my desire is to run the master there.

Some issues I had with other ideas:

• you cannot run k3s master natively on m1 atm via install script
• free hypervisor virtualization options on m1 are not so great and virtualbox which I have used in the past is a no go for M1.
  • I have used Parallels Desktop in the past which is good but I have found it interferes with my Line 6 Helix FX pedal (USB ports) and the helix editor causing drop outs so would rather avoid that as its annoy and not tenable.

Hence why K3D seemed like a great option as I have use it for a couple of years now for testing and it does run on Mac M1. Plus it is isolated as it runs in docker.

What have I tried?

1. Spin up a k3d master node using the following config file

Command used to spin up the cluster k3d cluster create demo --config k3d-config.yaml

apiVersion: k3d.io/v1alpha4
kind: Simple
servers: 1       # same as `--servers 1`
kubeAPI: # same as `--api-port myhost.my.domain:6445` (where the name would resolve to 127.0.0.1)
  host: "192.168.0.3" # important for the `server` setting in the kubeconfig
  hostIP: "0.0.0.0" # where the Kubernetes API will be listening on
  hostPort: "6445" # where the Kubernetes API listening port will be mapped to on your host system
image: rancher/k3s:v1.23.6-k3s1
token: superSecretToken
ports:
  - port: 8081:80 # same as `--port '8080:80@loadbalancer'`
    nodeFilters:
      - loadbalancer
options:
  k3s: # options passed on to K3s itself
    extraArgs: # additional arguments passed to the `k3s server|agent` command; same as `--k3s-arg`
      - arg: --tls-san=192.168.0.3
        nodeFilters:
          - server:*

2. Attempted to connect a K3S Client (no love)

export K3S_TOKEN="superSecretToken"
export K3S_URL="https://192.168.0.3:6445"
export INSTALL_K3S_CHANNEL='stable'

curl -sfL https://get.k3s.io | sh -

From the K3S node I can curl the K3D master.

curl -ks https://192.168.0.3:6445/ping
pong

However if we look at the agent service it appears a CA cert issue from this output.

It says to use the "full" token but I am not sure how to go about that using K3D (details in logs below)
So this is where I get stuck.

journalctl -u k3s-agent.service --since "10 minutes ago" - extract to show what that the systemctl does not, setup of load balancer

Jul 18 22:23:25 pi6-k3sn systemd[1]: Started Lightweight Kubernetes.
Jul 18 22:23:25 pi6-k3sn k3s[2010182]: time="2022-07-18T22:23:25+10:00" level=info msg="Acquiring lock file /var/lib/rancher/k3s/data/.lock"
Jul 18 22:23:25 pi6-k3sn k3s[2010182]: time="2022-07-18T22:23:25+10:00" level=info msg="Preparing data dir /var/lib/rancher/k3s/data/0519414e2a8313c0184273224578271d46c25d19a6299c8bc1780f4ec16b9a02"
Jul 18 22:23:33 pi6-k3sn k3s[2010182]: time="2022-07-18T22:23:33+10:00" level=info msg="Starting k3s agent v1.23.8+k3s2 (fe3cecc2)"
Jul 18 22:23:33 pi6-k3sn k3s[2010182]: time="2022-07-18T22:23:33+10:00" level=info msg="Running load balancer k3s-agent-load-balancer 127.0.0.1:6444 -> [192.168.0.3:6445]"
Jul 18 22:23:33 pi6-k3sn k3s[2010182]: time="2022-07-18T22:23:33+10:00" level=warning msg="Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation."
Jul 18 22:23:45 pi6-k3sn k3s[2010182]: time="2022-07-18T22:23:45+10:00" level=info msg="Module overlay was already loaded"
Jul 18 22:23:45 pi6-k3sn k3s[2010182]: time="2022-07-18T22:23:45+10:00" level=info msg="Module nf_conntrack was already loaded"

systemctl status k3s-agent.service

k3s-agent.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-07-18 22:23:25 AEST; 13min ago
       Docs: https://k3s.io
    Process: 2010178 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
    Process: 2010180 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 2010181 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 2010182 (k3s-agent)
      Tasks: 21
     Memory: 245.3M
        CPU: 16.390s
     CGroup: /system.slice/k3s-agent.service
             ├─2010182 /usr/local/bin/k3s agent
             └─2010271 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd

Jul 18 22:35:47 pi6-k3sn k3s[2010182]: time="2022-07-18T22:35:47+10:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: CA cert validation failed: Get \"https://127.0.0.1:6444/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 18 22:36:07 pi6-k3sn k3s[2010182]: time="2022-07-18T22:36:07+10:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: CA cert validation failed: Get \"https://127.0.0.1:6444/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 18 22:36:26 pi6-k3sn k3s[2010182]: W0718 22:36:26.353587 2010182 reflector.go:324] k8s.io/client-go@v1.23.8-k3s1/tools/cache/reflector.go:167: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": net/http: TLS handshake timeout
Jul 18 22:36:26 pi6-k3sn k3s[2010182]: I0718 22:36:26.353992 2010182 trace.go:205] Trace[768658237]: "Reflector ListAndWatch" name:k8s.io/client-go@v1.23.8-k3s1/tools/cache/reflector.go:167 (18-Jul-2022 22:36:16.351) (total time: 10002ms):
Jul 18 22:36:26 pi6-k3sn k3s[2010182]: Trace[768658237]: ---"Objects listed" error:Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": net/http: TLS handshake timeout 10002ms (22:36:26.353)
Jul 18 22:36:26 pi6-k3sn k3s[2010182]: Trace[768658237]: [10.002710004s] [10.002710004s] END
Jul 18 22:36:26 pi6-k3sn k3s[2010182]: E0718 22:36:26.354080 2010182 reflector.go:138] k8s.io/client-go@v1.23.8-k3s1/tools/cache/reflector.go:167: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": net/http: TLS handshake timeout
Jul 18 22:36:27 pi6-k3sn k3s[2010182]: time="2022-07-18T22:36:27+10:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: CA cert validation failed: Get \"https://127.0.0.1:6444/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 18 22:36:47 pi6-k3sn k3s[2010182]: time="2022-07-18T22:36:47+10:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: CA cert validation failed: Get \"https://127.0.0.1:6444/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 18 22:37:07 pi6-k3sn k3s[2010182]: time="2022-07-18T22:37:07+10:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: CA cert validation failed: Get \"https://127.0.0.1:6444/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"