Multi node support?

[kkimdev]

I tried microk8s locally and I really like it, it's super fast and solid. Though I was wondering if multi-node is out of scope for microk8s, or possibly can be supported in the future.

[marcoceppi]

From my understanding, this will only ever be single node. I've been working on a fork which adds multinode support on and off.

Hopefully the maintainers reconsider

[kkimdev]

@marcoceppi it's you again! 👍 Thanks for all the work and really looking forward to see them. With #233 and multi-node. I'm pretty sure microk8s will be easily the goto choice for Kubernetes on CI.

[ktsakalozos]

Hi @kkimdev

The multi node support is not yet on our roadmap. The reason is that we do not want to tempt users to run MicroK8s in a production environment. We need to first address security concerns that unfortunately often come at the expense of usability and maintenance costs that developers should not bother with.

@kkimdev how would you expect multi node support to look like? Would you expect to have multiple nodes running on the same machine (ie multiple kubelets) or would you expect to be able to "connect" many MicroK8s deployments and form a cluster?

@marcoceppi would you consider opening a PR? I cannot promise you we will merge it but it will offer a ground to start discussing this feature.

Thank you.

[kkimdev]

@ktsakalozos I totally understand that for the purpose of microk8s, multi node support could be an overkill and probably very low priority anyways. I would still happily use microk8s without that. Having said that, our main usecase is CI, and it's better if we can test a similar/same Kubernetes configuration that's used in the production, which includes things like nodeSelector/nodeAffinity that requires multi nodes to fully test. I can also imagine scenarios like exposing the same hostport from multiple nodes. Anyways, for us, multiple nodes running on the same machine would be very sufficient.

[JnRouvignac]

"multiple nodes running on the same machine" would be definitely useful. This would avoid ugliness like forcing to add if statements to helm charts so they would install on single node k8s (like minikube or microk8s).

[joneepenk]

+1 for multiple nodes in ci

[khteh]

When is the ETA for this support?

[ktsakalozos]

When is the ETA for this support?

There is no ETA yet

[lazzarello]

how would you expect multi node support to look like? Would you expect to have multiple nodes running on the same machine (ie multiple kubelets) or would you expect to be able to "connect" many MicroK8s deployments and form a cluster?

My use case for this feature is to add a Windows worker node to a Linux microk8s cluster, to test Windows specific docker images. This would be helpful because currently a Windows server cannot run a full cluster, only join a worker to a Linux cluster.

[khteh]

Windows worker "node"? Windows specific "docker inages"?

[ktsakalozos]

@lazzarello that is an interesting use case. Do you really need to have multiple nodes for this? I think that if you have kubelet+proxy+runtime+cni running on windows you should be able to stop the respective services in MicroK8s and use yours... I am probably too naive, the devil is in the details.

[Siegfriedk]

I'm working with k8s on a daily basis and like it very much.

Now i do have a PC 24/7 running at home and will install microk8s onto it to have a few services running @ home (Prometheus, temperature sensors, openhab).

Now what would be awesome to buy a small machine on anyprovider and install a microk8s node instance there and manage it through my PC at home. I also have an intel nuc at home and would like to utilize it also as a Node to play around with affinity and daemonsets.

[lazzarello]

Do you really need to have multiple nodes for this? I think that if you have kubelet+proxy+runtime+cni running on windows you should be able to stop the respective services in MicroK8s and use yours

Windows Server 2019 is the only supported OS for Kubernetes on Windows and is limited to running workers only. No apiserver or scheduler, etc. The server must join a Linux cluster. This may change but here we are...

@khteh A Windows worker node. I need to schedule CI jobs which run Windows EXE binaries.

[sr229]

+1 on this! Our team has many headaches regarding K3s (thank you for hardcoding your CNI configs Rancher) and this will help us run Kubernetes without the grudgery of manually installing and updating it.

Our use case for this is basically running a service behind a LXC cluster which should allow us to have a more secure way to have multi-nodes

[ktsakalozos]

@sr229 since you are familiar with LXC you may want to look at the local (LXC) deployment of Charmed Distribution of Kubernetes (CDK) https://www.ubuntu.com/kubernetes/docs/install-local . CDK is from the same team as MicroK8s and is production grade.

[torstenwerner]

I occasionally want to add a 2nd worker node with lots of memory for a limited time span. Think of 2 nodes during rush hour and 1 smaller node during the rest of the time. Having a single node with optional dual node support would be perfect for me. ;)

[sihrc]

Very interested in this feature add as well. We're currently hitting the 110 pod limit per node, and based on what we've seen in the existing issues around it, it's an implementation out of our control. If we could have multiple "nodes" in one machine, we can effectively work around this problem and better mirror our normal multinode setups.

[ktsakalozos]

We have started working on this feature. For anyone interested in lending a hand here is the branch https://github.com/ubuntu/microk8s/tree/feature/clustering

[StableCube]

Thank you for adding this feature. I have a need to simulate production with nodes that have special tags and taints that is just not possible with a single node and has forced me into considering a local bare metal cluster for development or a paying for a dev cluster at my datacenter. Both are not ideal.

As a bit of an aside, a cool feature would be the ability to add VirtualBox images as nodes to streamline the process of adding nodes. I am not sure of the feasibility of this or if it would be out of scope for micok8s but it sure would be cool.

[sr229]

@StableCube you can always use Ansible to setup Microk8s to do that for you

[rnestler]

Isn't this possible already? According to https://microk8s.io/docs/clustering it is.

[ktsakalozos]

This feature has been available since v1.16. Please see https://microk8s.io/docs/clustering and open separate issues for any features you want to see. Thank you for your patience and feedback.