Running a one node K3S cluster on a Linux VM running on Hyper-V should not be that hard. Adding a second node to the cluster, is also easy. I fond some resources on how to run k3s in WSL2. But then I wanted to have a two node K3S cluster running on WSL2 and this is how you can make it hapen.
- K3S is a certified kubernetes distro for edge and other scenarios
- WSL2 runs Linux on windows, nicer and more integrated than a traditional vm on hyper-v
- Install Rio on K3S running in WSL2 on Windows
- k3d - K3S in docker
- Windows Terminal - great experience
- docker export - export running container to rootfs
- Windows 10 2004 OS Build 19041.xxx or newer with WSL installed
- A Linux system where docker is installed and running
- Ubuntu installed from the Microsoft Store
- kubectl installed on your system
-
open Windows Terminal and connect to your WLS Ubuntu instance
-
connect to your Linux system that runs docker using ssh ; I am using a VM running Ubuntu 16.04.6 LTS
-
create a working folder, to be used for creating the K3S image for to be used in WSL2
mkdir k3s-wsl2
-
prepare the files to created the K3S image and container
profile
andDockerfile
cat <<EOF > ./profile export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/bin/aux mkdir /sys/fs/cgroup/systemd mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd EOF
cat <<EOF > ./Dockerfile FROM rancher/k3s:v1.17.4-k3s1-amd64 COPY ./profile /etc/profile EOF
-
create the K3S image, start a container and export container’s filesystem as a tar archive
docker build . --tag wsl2-k3s-v1.17.4-k3s1 docker run --name wsl2-k3s wsl2-k3s-v1.17.4-k3s1 docker export --output wsl2-k3s-v1.17.4-k3s1.tar wsl2-k3s
-
copy the tar file,
wsl2-k3s-v1.17.4-k3s1.tar
, to your local Windows system, e.gC:\wsl2\
-
in Windows Terminal open a PowerShell instance
-
change the working directory to
c:\wsl2
-
create a folder, where the files of the k3s-node-0 WSL2 instace will be saved
PS C:\>C:\wsl2\ mkdir C:\wsl2\k3s-node-0
-
import the K3S image, that was copied eralier from your Linux box, as a WSL instance version 2
PS C:\>C:\wsl2\ wsl --import k3s-node-0 C:\wsl2\k3s-node-0\ C:\wsl2\wsl2-k3s-v1.17.4-k3s1.tar --version 2
-
check created instance - it will use version 2
PS C:\>C:\wsl2\ wsl -l -v NAME STATE VERSION * Ubuntu Running 1 k3s-node-0 Stopped 2 PS C:\>
-
start the K3S instance
PS C:\>C:\wsl2\ wsl -d k3s-node-0
-
at the new promp, inside the k3s instance, check k3s version, add an ID to
/etc/machine-id
and start the K3S server/ # / # k3s --version k3s version v1.17.4+k3s1 (3eee8ac3) / # / # od -x /dev/urandom | head -1 | awk '{OFS="-"; print $2$3,$4,$5,$6,$7$8$9}' > /etc/machine-id / # / # nohup k3s server & / #
-
several seconds later, K3s is running
/ # kubectl get nodes NAME STATUS ROLES AGE VERSION pc0qap6g Ready master 10m v1.17.4+k3s1 / # / # kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system local-path-provisioner-58fb86bdfd-wqcll 1/1 Running 0 17m kube-system coredns-6c6bb68b64-xz8pd 1/1 Running 0 17m kube-system metrics-server-6d684c7b5-hkxfp 1/1 Running 0 17m kube-system helm-install-traefik-sr67m 0/1 Completed 0 17m kube-system svclb-traefik-nppmc 2/2 Running 0 14m kube-system traefik-7b8b884c8-glxdp 1/1 Running 0 14m / #
-
get the IP address of the K3S server and the token; we need this to add the second node to the cluster
/ # ip addr sho eth0 scope global|grep inet|awk '{print $2}'|awk -F'/' '{print $1}' 172.23.120.165 / # / # cat /var/lib/rancher/k3s/server/node-token / # K1034daef0594ec255977adc416af8c57bf2eb914167fc3ee241ea134a246621c6c::server:175d0215b69ffacc9bd03982ae8a3bf9 / #
-
open PowerShell with Admin rights
-
create a new local user and assign it a password
PS C:\> $Password = Read-Host -AsSecureString ********* PS C:\> New-LocalUser -Name "k3s-node-1" -AccountNeverExpires -Password $Password Name Enabled Description ---- ------- ----------- k3s-node-1 True PS C:\>
-
next, we will run PowerShell in the new user context to create the folder to install the new WSL2 instance, and import the K3S image
PS C:\> PS C:\> $Credential = Get-Credential k3s-node-1 PS C:\> start-process powershell -credential $Credential -LoadUserProfile -ArgumentList "mkdir c:\wsl2\k3s-node-1;exit" PS C:\> start-process powershell -credential $Credential -LoadUserProfile -ArgumentList "wsl --import k3s-node-1 C:\wsl2\k3s-node-1\ C:\wsl2\wsl2-k3s-v1.17.4-k3s1.tar --version 2" PS C:\> start-process powershell -credential $Credential -LoadUserProfile -ArgumentList "wsl -l -v;start-sleep 10;exe" PS C:\>
-
we can now start the new WSL2 instance and check the verison of the K3S; a new window will pop up and close in 10sec
PS C:\> start-process powershell -credential $Credential -LoadUserProfile -WorkingDirectory c:\wsl2 -ArgumentList "wsl -d k3s-node-1 k3s --version;start-sleep 10;exit"
-
before starting
k3s agent
we need to create a soft link formount
in/bin
folder in the interactive mode/bin/aux/
is part of the$PATH
but not when runningwsl ---exec $PATH
PS C:\> Start-Job -Credential $Credential -Name k3s-node-1 -ScriptBlock {wsl -d k3s-node-1 ln -s /bin/aux/mount /bin/mount} Id Name PSJobTypeName State HasMoreData Location Command -- ---- ------------- ----- ----------- -------- ------- 19 k3s-node-1 BackgroundJob Running True localhost wsl -d k3s-node-1 ln -...
-
we prepare the arguments and start the
k3s agent
usingStart-Job
this is needed becasewsl --exe
does not accept special characters like&
PS C:\> $script = {wsl -d k3s-node-1 -e k3s agent --server https://172.23.120.165:6443 --token K1034daef0594ec255977adc416af8c57bf2eb914167fc3ee241ea134a246621c6c::server:175d0215b69ffacc9bd03982ae8a3bf9 --with-node-id} PS C:\> PS C:\> Start-Job -Credential $Credential -Name k3s-node-1 -ScriptBlock $script Id Name PSJobTypeName State HasMoreData Location Command -- ---- ------------- ----- ----------- -------- ------- 21 k3s-node-1 BackgroundJob Running True localhost wsl -d k3s-node-1 -e k...
-
in Windows Terminal open a PowerShell instance
-
check using
wsl
the status of the cluster and nodesPS C:\> wsl -l -v NAME STATE VERSION * Ubuntu Running 1 k3s-node-0 Running 2 PS C:\> PS C:\> wsl -d k3s-node-0 kubectl get nodes NAME STATUS ROLES AGE VERSION pc0qap6g-2ea44fa8 Ready <none> 63m v1.17.4+k3s1 172.23.114.210 <none> Unknown 4.19.84-microsoft-standard containerd://1.3.3-k3s2 pc0qap6g Ready master 5h20m v1.17.4+k3s1 172.23.120.165 <none> Unknown 4.19.84-microsoft-standard containerd://1.3.3-k3s2 PS C:\> PS C:\> wsl -d k3s-node-0 kubectl top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% pc0qap6g 133m 3% 684Mi 5% pc0qap6g-2ea44fa8 19m 0% 109Mi 0% PS C:\> PS C:\> wsl -d k3s-node-0 kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system local-path-provisioner-58fb86bdfd-wqcll 1/1 Running 0 5h18m kube-system coredns-6c6bb68b64-xz8pd 1/1 Running 0 5h18m kube-system metrics-server-6d684c7b5-hkxfp 1/1 Running 0 5h18m kube-system helm-install-traefik-sr67m 0/1 Completed 0 5h18m kube-system svclb-traefik-nppmc 2/2 Running 0 5h15m kube-system traefik-7b8b884c8-glxdp 1/1 Running 0 5h15m kube-system svclb-traefik-mntpx 2/2 Running 0 24m PS C:\>
-
extract the kubeconfig file from the cluster, replace 127.0.0.1 with the eth0 ip address of the k3s server config file saved to c:\wsl2\k3s.yaml
PS C:\wsl2> (wsl -d k3s-node-0 cat /etc/rancher/k3s/k3s.yaml) -replace '127.0.0.1',(wsl -d k3s-node-0 ip addr show eth0|?{$_ -match "inet "}).split()[5].split("/")[0] > c:\wsl2\k3s.yaml
-
use local
kubectl
and check cluster resourcePS C:\wsl2> PS C:\wsl2> kubectl --kubeconfig=c:\wsl2\k3s.yaml get nodes NAME STATUS ROLES AGE VERSION pc0qap6g-2ea44fa8 Ready <none> 83m v1.17.4+k3s1 pc0qap6g Ready master 5h41m v1.17.4+k3s1 PS C:\wsl2> PS C:\wsl2> kubectl --kubeconfig=c:\wsl2\k3s.yaml top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% pc0qap6g 125m 3% 686Mi 5% pc0qap6g-2ea44fa8 20m 0% 109Mi 0% PS C:\wsl2> PS C:\wsl2> kubectl --kubeconfig=c:\wsl2\k3s.yaml get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system local-path-provisioner-58fb86bdfd-wqcll 1/1 Running 0 5h41m kube-system coredns-6c6bb68b64-xz8pd 1/1 Running 0 5h41m kube-system metrics-server-6d684c7b5-hkxfp 1/1 Running 0 5h41m kube-system helm-install-traefik-sr67m 0/1 Completed 0 5h41m kube-system svclb-traefik-nppmc 2/2 Running 0 5h38m kube-system traefik-7b8b884c8-glxdp 1/1 Running 0 5h38m kube-system svclb-traefik-mntpx 2/2 Running 0 47m PS C:\wsl2>
still looking for a way to get the rootfs of an image , without the need to have a Linux /Windows computer that is running docker point 3.6 - still need to find a way to push a guid to /etc/machine-id , in a one-liner