Skip to content

Latest commit

 

History

History
255 lines (194 loc) · 9.75 KB

k0s-in-docker.md

File metadata and controls

255 lines (194 loc) · 9.75 KB
Raw

Run k0s in Docker

You can create a k0s cluster on top of Docker.

Prerequisites

You will require a Docker environment running on a Mac, Windows, or Linux system.

Container images

The k0s OCI images are published to both Docker Hub and GitHub Container registry. For simplicity, the examples given here use Docker Hub (GitHub requires separate authentication, which is not covered here). The image names are as follows:

  • docker.io/k0sproject/k0s:v{{{extra.k8s_version}}}-k0s.0
  • ghcr.io/k0sproject/k0s:v{{{extra.k8s_version}}}-k0s.0

Note: Due to Docker's tag validation scheme, - is used as the k0s version separator instead of the usual +. For example, k0s version v{{{extra.k8s_version}}}+k0s.0 is tagged as docker.io/k0sproject/k0s:v{{{extra.k8s_version}}}-k0s.0.

Start k0s

1. Run a controller

By default, running the k0s OCI image will launch a controller with workloads enabled (i.e. a controller with the --enable-worker flag) to provide an easy local testing "cluster":

docker run -d --name k0s-controller --hostname k0s-controller \
  -v /var/lib/k0s -v /var/log/pods `# this is where k0s stores its data` \
  --tmpfs /run `# this is where k0s stores runtime data` \
  --privileged `# this is the easiest way to enable container-in-container workloads` \
  -p 6443:6443 `# publish the Kubernetes API server port` \
  docker.io/k0sproject/k0s:v{{{extra.k8s_version}}}-k0s.0

Explanation of command line arguments:

  • -d runs the container in detached mode, i.e. in the background.
  • --name k0s-controller names the container "k0s-controller".
  • --hostname k0s-controller sets the hostname of the container to "k0s-controller".
  • -v /var/lib/k0s -v /var/log/pods creates two Docker volumes and mounts them to /var/lib/k0s and /var/log/pods respectively inside the container, ensuring that cluster data persists across container restarts.
  • --tmpfs /run TODO
  • --privileged gives the container the elevated privileges that k0s needs to function properly within Docker. See the section on adding additional workers for a more detailed discussion of privileges.
  • -p 6443:6443 exposes the container's Kubernetes API server port 6443 to the host, allowing you to interact with the cluster externally.
  • docker.io/k0sproject/k0s:v{{{ extra.k8s_version }}}-k0s.0 is the name of the k0s image to run.

By default, the k0s image starts a k0s controller with worker components enabled within the same container, creating a cluster with a single controller-and-worker node using the following command:

{% include "../Dockerfile" start="# Start CMD" end="# End CMD" %}

Alternatively, a controller-only node can be run like this:

docker run -d --name k0s-controller --hostname k0s-controller \
  --read-only `# k0s won't write any data outside the below paths` \
  -v /var/lib/k0s `# this is where k0s stores its data` \
  --tmpfs /run `# this is where k0s stores runtime data` \
  --tmpfs /tmp `# allow writing temporary files` \
  -p 6443:6443 `# publish the Kubernetes API server port` \
  docker.io/k0sproject/k0s:v{{{extra.k8s_version}}}-k0s.0 \
  k0s controller

Note the addition of k0s controller to override the image's default command. Also note that a controller-only node requires fewer privileges.

2. (Optional) Add additional workers

You can add multiple worker nodes to the cluster and then distribute your application containers to separate workers.

  1. Acquire a join token for the worker:

    token=$(docker exec k0s-controller k0s token create --role=worker)

  2. Run the container to create and join the new worker:

    docker run -d --name k0s-worker1 --hostname k0s-worker1 \
  -v /var/lib/k0s -v /var/log/pods `# this is where k0s stores its data` \
  --tmpfs /run `# this is where k0s stores runtime data` \
  --privileged `# this is the easiest way to enable container-in-container workloads` \
  docker.io/k0sproject/k0s:v{{{extra.k8s_version}}}-k0s.0 \
  k0s worker $token

    Alternatively, with fine-grained privileges:

    docker run -d --name k0s-worker1 --hostname k0s-worker1 \
  -v /var/lib/k0s -v /var/log/pods `# this is where k0s stores its data` \
  --tmpfs /run `# this is where k0s stores runtime data` \
  --security-opt seccomp=unconfined \
  -v /dev/kmsg:/dev/kmsg:ro --device-cgroup-rule='c 1:11 r' \
  --cap-add sys_admin \
  --cap-add net_admin \
  --cap-add sys_ptrace \
  --cap-add sys_resource \
  docker.io/k0sproject/k0s:v{{{extra.k8s_version}}}-k0s.0 \
  k0s worker "$token"

    Notes on the security-related flags:

    • --security-opt seccomp=unconfined is required for runc to access the session keyring.
    • -v /dev/kmsg:/dev/kmsg:ro --device-cgroup-rule='c 1:11 r' allows reading of /dev/kmsg from inside the container. The kubelet's OOM watcher uses this.

    Notes on Linux capabilities:

    • CAP_SYS_ADMIN allows for a variety of administrative tasks, including mounting file systems and managing namespaces, which are necessary for creating and configuring nested containers.
    • CAP_NET_ADMIN allows manipulation of network settings such as interfaces and routes, allowing containers to create isolated or bridged networks, and so on.
    • CAP_SYS_PTRACE allows to inspect and modify processes, used to monitor other containers in a nested environment.
    • CAP_SYS_RESOURCE allows containers to override resource limits for things like memory or file descriptors, used to manage and adjust resource allocation in nested container environments.

    Note that more privileges may be required depending on your cluster configuration and workloads.

    Repeat this step for each additional worker node and adjust the container and host names accordingly. Make sure that the workers can reach the controller on the required ports. If you are using Docker's default bridged network, this should be the case.

3. Access your cluster

a) Using kubectl within the container

To check cluster status and list nodes, use:

docker exec k0s-controller k0s kubectl get nodes

b) Using kubectl locally

To configure local access to your k0s cluster, follow these steps:

  1. Generate the kubeconfig:

    docker exec k0s-controller k0s kubeconfig admin > ~/.kube/k0s.config

  2. Update kubeconfig with Localhost Access:

    To automatically replace the server IP with localhost dynamically in ~/.kube/k0s.config, use the following command:

    sed -i '' -e "$(awk '/server:/ {print NR; exit}' ~/.kube/k0s.config)s|https://.*:6443|https://localhost:6443|" ~/.kube/k0s.config

    This command updates the kubeconfig to point to localhost, allowing access to the API server from your host machine

  3. Set the KUBECONFIG Environment Variable:

    export KUBECONFIG=~/.kube/k0s.config

  4. Verify Cluster Access:

    kubectl get nodes

c) Use Lens

Access the k0s cluster using Lens by following the instructions on how to add a cluster.

Use Docker Compose (alternative)

As an alternative you can run k0s using Docker Compose:

{% include "compose.yaml" %}

Known limitations

No custom Docker networks

Currently, k0s nodes cannot be run if the containers are configured to use custom networks (for example, with --net my-net). This is because Docker sets up a custom DNS service within the network which creates issues with CoreDNS. No completely reliable workaounds are available, however no issues should arise from running k0s cluster(s) on a bridge network.

Next Steps