Local Docker Talos with Cilium CNI

[maxwnewcomer]

Hello, I've been trying to run a local talos cluster with cilium and I am running into some issues getting it up and going.
I am currently on

• Docker 27.3.1
• Talos 1.8.3
• MacOS 15.1.1 (24B91) m2

The best I've gotten so far was running

talosctl cluster create --with-kubespan --config-patch @patch.yml

with patch (current path ./patch.yml)

cluster:
  network:
    cni:
      name: none
  proxy:
    disabled: true

This times out with

◱ waiting for all k8s nodes to report ready: some nodes are not ready: [talos-default-controlplane-1 talos-default-worker-1]
context deadline exceeded

When I try to install cilium with helm before that i get:

helm install \
    cilium \
    cilium/cilium \
    --version 1.15.6 \
    --namespace kube-system \
    --set ipam.mode=kubernetes \
    --set kubeProxyReplacement=false \
    --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
    --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
    --set cgroup.autoMount.enabled=false \
    --set cgroup.hostRoot=/sys/fs/cgroup
    --set k8sServiceHost=localhost \
    --set k8sServicePort=7445
Error: INSTALLATION FAILED: Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp [::1]:8080: connect: connection refused

That felt unintuitive to me, but I thought I'd give it a shot because the docs said that:

After applying the machine config and bootstrapping Talos will appear to hang on phase 18/19 with the message: retrying error: node not ready. This happens because nodes in Kubernetes are only marked as ready once the CNI is up. As there is no CNI defined, the boot process is pending and will reboot the node to retry after 10 minutes, this is expected behavior.

I feel like I'm missing something obvious and would really appreciate any help!

[smira]

It's right, the health check won't pass until the CNI is up. So you can skip the healthcheck. But you need to fetch cluster kubeconfig (it won't happen automatically as the health doesn't pass).

Once the apid health passes, you can do talosctl -n 10.5.0.2 khbeconfig and proceed to helm install Cilium

[maxwnewcomer]

That was exactly what I was looking for! Thank you so much.

As a consolidated solution for anyone looking at this later:

Talos Local w Cilium CNI

1. Create a file patch.yml

cluster:
  network:
    cni:
      name: none
  proxy:
    disabled: true

2. Run talosctl cluster create --config-patch @patch.yml --skip-k8s-node-readiness-check
3. After you see waiting for apid to be ready: OK you can run talosctl -n 10.5.0.2 kubeconfig (swapping out 10.5.0.2 if you changed your controlplane ip)

• Depending on what the cluster create status and how many times you've ran talosctl cluster create you may need to overwrite the config when running talosctl -n 10.5.0.2 kubeconfig

4. To install the latest version of the Cilium CNI we can run

helm install \                                                                   
    cilium \
    cilium/cilium \
    --version 1.16.4 \
    --namespace kube-system \
    --set ipam.mode=kubernetes \
    --set kubeProxyReplacement=false \
    --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
    --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
    --set cgroup.autoMount.enabled=false \
    --set cgroup.hostRoot=/sys/fs/cgroup \
    --set k8sServiceHost=localhost \
    --set k8sServicePort=7445

5. Check cilium with cilium status --wait and you should see everything healthy.

Untested additional commands

• Cilium recommends running the following after installation, not sure on if this is truly necessary with the above steps

kubectl get pods --all-namespaces -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,HOSTNETWORK:.spec.hostNetwork --no-headers=true | grep '<none>' | awk '{print "-n "$1" "$2}' | xargs -L 1 -r kubectl delete pod

I'm still digging into getting cilium connectivity test working, but at least now everything with cilium seems healthy!

Thank you so much for taking the time to help out.