Future of k0s, a CNCF project?

[jnummelin]

Hey all,

The team k0s is thinking of submitting k0s as a CNCF Sandbox project. We've not yet made the final go/no-go decision but in order to gather more "evidence" and prep for the potential application we're collecting list of users and adopters that we could utilise for the submission as "supporters".

So if you're a k0s user, which you probably are if you're reading this issue 😄 , please add some details as a reply. Would be awesome if you can share your contact info, some words how you use k0s and in which kind of use case you use it on. If you do not wish to share that kind of details in public you can also reach out to me via email jnummelin [at] mirantis.com. If nothing else, please at least show your support using emoji reactions on this. 😄

What would it mean to have k0s as a CNCF project? (i.e. what's in it for you)

IF k0s would get accepted as a CNCF project we'd be able to create even stronger foundation for the project and bolster both the adoption and community involvement. In the end that means better foundation for YOU to build upon.

[laghoule]

Using it for my home cluster, and evaluating it for OnPrem usage at work.

[ferama]

We started using it in production for very simple scenarios (single node clusters). We have also some multi node development environments and there I feel that I don't have the full control of what is happening on the control plane nodes (it's why I opened this one #3247 for example).

We love the idea of k0s and we also (almost) love k0sctl. I would like to use it for bigger clusters and maybe to go on and use it for bigger production scenarios. We are not there yet, I hope we are some day :)

I would love to read more about experiences on running k0s for bigger clusters, but I'm not able to find almost anything.
A bigger community could help for sure.

Thanks

[laghoule]

An official k0s terraform provisioner should definitively give a good boost to enterprise adoption.

[Starttoaster]

My home 3 node HA cluster runs k0s, deployed with k0sctl. We're also evaluating it for onprem use at my workplace. In the future of the k0s project, I would like to see documentation on running an HA API server load balancer. Currently, the documentation only shows how to set up a single HAProxy instance to load balance the API server.

[CmdrSharp]

We are trialling it as a replacement for clusters currently run via Google Anthos. It offers a similarly neat upgrade experience based on our testing so far. We are still in early testing, but like the concept and if all goes well, we'll do some production-grade trials this autumn.

[Skaronator]

I'm using k0s in my homelab as well. It works great, especially since it's pretty lightweight and doesn't include that many services out of the box (unlike k3s).

Also using it on prem at work. Works just as good. I love k0sctl and the k0sctl.yaml. Works really great for versioning in git. Another feature that k3s doesn't have.

---

I would love to have 2 things to improve:

• Faster release of major version. k3s is sometimes 3 or 4 weeks ahead: First 1.27 release of k3s was April 27 while k0s was on May 23.
• Better handling of changes within the k0sctl.yaml. e.g.
  • allow disabling or re-enabling of components (like metrics server)
  • allow changing role of hosts, sans or externalAddress
  • allow changing of network configuration
  • if a change is not possible with the currently active cluster, then just say that to the user. k0sctl should offer a way to migrate to the new configuration -> e.g. backup the current cluster, reset and run apply. All within the same operation.

[Rohmilchkaese]

We implemented it in Work as Production Cluster! Would definitely support

[jsalgado78]

+1

We've three production clusters running k0s

We implemented it in Work as Production Cluster! Would definitely support

[pschichtel]

I'm using it for my home lab and at work for both development and production clusters on prem. I'm using k0sctl exclusively for all deployments. In the last 3 weeks I introduced 2 additional teams to k0s.

You can find my contact details on my website (linked in my profile).

[djds]

I think this would be really a good move for the health of the project and also demonstrate a commitment to the community from Mirantis. As many high-profile open source projects have recently switched to a source available, non open source license, it would significantly disambiguate the future of any investments community members or organizations make to the project knowing that k0s would remain Apache 2.0. It's simply not viable for many businesses to invest in core infrastructure that might have the rug pulled out from under it at any point in the future, and foundation ownership of the project is a strong bulwark against that risk.

k0s is a great Kubernetes distribution: I run it on a 8 node cluster with 7 Raspberry Pi CM4s and an old Thinkpad to provide one node with x86_64 support, all running Fedora CoreOS as the host OS on my home lab. I work with many of the managed K8s distributions from the hyperscalers at my $dayjob, but k0s is really useful for testing Kubernetes workloads on prem and for hosting all of the cloud native applications that I have to deal with in a professional setting.

Thanks for all of the work that the team has put in! It's nice to be able to get up and running with a simple vanilla Kubernetes distribution with good security defaults.

[vyas-n]

I'm currently using k0s as a part of a Dev cluster at my company.

I'm having it soak for a period of time to prove stability before presenting it to my company as a potential platform for Production.

Having k0s be a CNCF Project will make us much more confident in its production grade use and be able to contribute upstream for bugs we find or docs we'd like for reference later.

[marccampbell]

We are building around the k0s project, and would love to see k0s become a CNCF project. Our use case at Replicated is to enable an embedded appliance experience as part of a distribution mechanism.

If the k0s project was a CNCF project, we'd be able to have more confidence that the license will continue to allow our use case, and being able to participate in the open governance of the project.

[alexdub37]

Hi !
I use k0s on-prem on multiple clusters.
I really like the simplicity. I also operate Openshift and Rancher clusters (as a sysadmin).
They are bloated, and that's where k0s shines.
It follows the unix philosophy of doing one thing, and doing it right.

k0s is a little bit of an outsider but has nothing less powerful than giant names.
Yes, it is vanilla. That's why I prefer it to OCP/Rancher.
No unlimited built-in feature list. Just Kubernetes well-packaged in a bin file.

Then, do what you want with it. Rook, Tekton, ArgoCD, OPA Gatekeeper, ...
Everything is one Helm chart away.

[sombriks]

so far k0s looks simple as kind or k3s, still trying to figure out how they compare to each other.

i need to test things before apply them on eks, therefore k0s looks the tool for the job.

[ams0]

Been using it for a while at home and now on Hetzner bare metal, I recommend it to everyone I speak to. Not huge marketing machine behind, which is a plus. I'm a CNCF Ambassador and I'd love to sponsor/promote the incubation efforts.

[djds]

One of the best parts is that you can use whatever CNI you want (although it's nice to include both Kuberouter and Calico as default options). By setting networkProvider: custom, I was able to get a full Cilium setup working, which I use for my homelab but also to help clients debug networking issues by having a quick environment to stand up. Other small Kubernetes distributions use non-default command-line flags or otherwise implement some DSL that means testing out vanilla Kubernetes features is not as easy.

[binacloud]

Hi,

We've deployed 2 production clusters with k0s and have 100% uptime for 2 months already. Will like to push this to more of my clients.

In these day and age, we need a robust yet cost-friendly solution to run workloads on-premise and I believe k0s can give the bigger players a run for their money !!

[randybias]

Just curious... why did folks choose k0s over another distribution like k3s?

[pschichtel]

for me it's been four reasons:

• close to zero distro dependencies (I've been running this on an ancient centos without issue)
• sane defaults close to "vanilla" k8s
• k0sctl
• the codebase is approachable, the team behind it is responsive and no CLA to sign my rights away; so it's a project that I would contribute to (which I did).

[randybias]

Thank you. That is a very clear and concise response.  We really appreciate it. Just reading your response is great as it is exactly what we intended. The validation is amazing. Thank you,--Randy*** Sent from mobile, pls forgive tpyoes ***On Aug 26, 2024, at 23:36, Phillip Schichtel ***@***.***> wrote: for me it's been four reasons: close to zero distro dependencies (I've been running this on an ancient centos without issue) sane defaults close to "vanilla" k8s k0sctl the codebase is approachable, the team behind it is responsive and no CLA to sign my rights away; so it's a project that I would contribute to (which I did). —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[leleobhz]

for me it's been four reasons:

• close to zero distro dependencies (I've been running this on an ancient centos without issue)
• sane defaults close to "vanilla" k8s
• k0sctl
• the codebase is approachable, the team behind it is responsive and no CLA to sign my rights away; so it's a project that I would contribute to (which I did).

Same reasons here with a plus: It's easy to configure k0s.conf/k0sctl.conf to cluster do things that it's normally not possible with k3s, as example my attempts to run k0s cluster within Rpi3 boards.

[randybias]

We are going to apply to the CNCF for k0s to enter as a sandbox project. Please chime in here with any additional thoughts, supporting data, or similar that can help us with explaining to the CNCF the unique value of k0s.

cncf/sandbox#125

[jnummelin]

We'd be more than happy to see PRs adding everyones use case to the list of adopters: https://github.com/k0sproject/k0s/blob/main/ADOPTERS.md

[danielr1996]

I'm currently using it for my home cluster, demo workshops for kubernetes that go more in depth that just using minikube and evaluate it for a production minecraft cluster. In the future i'd like to see a "official/community maintained" terraform/opentofu provider (although the ability to pipe terraform output to k0sctl stdin also works really well, further effort in k0smotron and clusterapi integration and maybe an sshless bootstrap of nodes with cloud init.

[marvin-hansen]

I am currently evaluating k0s and the CNCF application already ticks off the biggest topic on my list: Vendor neutral governance and continuity.

Last year, I donated a project of mine to the AI Linux Foundation and, looking back, it gives so much credibility and confidence that it was easily the best decision I've ever made.

On dealing with the foundation:

• It's good to ask for the onboarding checklist and work through it line by line.
• Ensure all the security requirements are in place
• You probably want to document the "hit by the bus" contingency plan i.e. what to do if the key maintainer with the GitHub admin password got hit by a bus so you better have a plan somewhere. And let's hope that plan is never used.

From memory, the process from application to decision may take up to three months and usually requires a presentation to the TA committee that votes whether to accept the project. The TA committee is the technical advisor committee comprised of industry members so these guys are usually extremely competent in what they do. During the presentation the biggest question really is how does your project differentiates itself and how it supports the CNCF mission?

In my project, that was relatively easy to answer as it is the first hypergeometric computational causality library written in Rust and obviously the foundation wants to be at the forefront of innovation.

For k0s, you would have to be specific how exactly it's different from k3s and how exactly it promotes cloud native. I don't think that just "simpler than k8s" alone would make the cut. Rather, I would brainstorm with the community and put forward a compelling innovation vision to drive home the point that your project is going to be the next cool thing.

For example:

• native support for WASM modules for high density isolated workloads. Just saying there is a potential differentiator to add over time.

• Some synergy with crossplane i.e. cluster in cluster provisioning / virtualization. Afaik, CNCF members quite often have an under utilization problem due to clients over provisioning clusters about 70% of the time. Things could be different if clients could natively create new virtual clusters from within their main cluster and that's exactly where cross plane shines. Also, Flux & Argo could use cross plane to spin up a virtual cluster before deploying an application to it

• Probably something Ai related would be cool , i.e. Ai LLM that genetates yaml configuration files from a prompt or something, but I think you should be perfectly fine without it.

These are just some random ideas, other people have hopefully some more ideas and at some point you can make a list, rank everything from crazy to sensible and select the top three to put on the roadmap. The foundation doesn't check if you implement everything by the dot, but they do want to know where the project is going.

As I said, there is the logistics of the onboarding with checklists, and there is the TA meeting where you present the project. Saying how k0s differe is half the message, and saying where it is going is the other half.

So happy to see k0s applying to CNCF.

[cloudcodger]

Currently managing 20+ on-prem clusters running k0s.

Had multiple issues with k0sctl not being very idempotent and ultimately created my own Ansible Role for deploying a cluster to multiple systems. I did see an Ansible Playbook for installing k0s where I feel a better approach is to provide one or more Ansible Roles with examples of how to use them in a playbook of your own.

One big issue is the lack of ability to do anything with the control plane. By design, it is great to have it separated such that worker nodes cannot as easily inject issues, but this leaves out easy methods of diagnosing problems on the control plane nodes. For example, I can create a 6 node cluster (3 control nodes and 3 worker nodes) and not load up any pods or install any non-default services, then let it sit rather idle and in less than 24 hours, one or more of the control nodes has a process start taking up 100% of one CPU core. The system running the control node, will then no longer be accessible via SSH. As if the networking layer of the host has failed in some way. Because I can no longer access the host, the only option is to power cycle it. A check of the logs after the reboot gives no indication of what failed or hung or caused a never ending loop (usually the thing that causes a single core to jump to 100% usage. I have not yet been able to resolve this issue.

Still trying to figure out how to easily enable HA of the control plane. For some reason the documentation appears to be incomplete or I have missed some critical part of setting this up.

Most documentation I find is centered around getting an initial (and very limited) configuration up and running. Comprehensive documentation on creating a fully HA configuration, including multiple control nodes accessible both externally and internal to the cluster to host a production environment is limited and is much needed for those wanting to use k0s in a production environment.

[CmdrSharp]

Still trying to figure out how to easily enable HA of the control plane. For some reason the documentation appears to be incomplete or I have missed some critical part of setting this up.

Most documentation I find is centered around getting an initial (and very limited) configuration up and running. Comprehensive documentation on creating a fully HA configuration, including multiple control nodes accessible both externally and internal to the cluster to host a production environment is limited and is much needed for those wanting to use k0s in a production environment.

In-cluster load balancing (and HA) is documented: https://docs.k0sproject.io/stable/nllb/

External load balancing is as well: https://docs.k0sproject.io/stable/high-availability/#load-balancer

What do you feel is missing here?

[cloudcodger]

Still trying to figure out how to easily enable HA of the control plane. For some reason the documentation appears to be incomplete or I have missed some critical part of setting this up.

In-cluster load balancing (and HA) is documented: https://docs.k0sproject.io/stable/nllb/

I configured a test cluster using this page and yet I still have

sage@sage:~/work % ssh ubuntu@work1 sudo grep server /var/lib/k0s/kubelet.conf
    server: https://192.168.6.21:6443

This is the IP address of the ctrl1 node and not a load balanced IP. BTW, all the worker nodes have this. I don't see anywhere that lets me know how to set it up to be a load balanced IP address for the control plane. It is very possible I just don't understand or missed where it says this and how to set it.

[onedr0p]

I don't want to sound like a pain but for people like me that are following this issue for updates can we keep it on topic? Thanks 🙏🏼

[cloudcodger]

Still trying to figure out how to easily enable HA of the control plane. For some reason the documentation appears to be incomplete or I have missed some critical part of setting this up.

External load balancing is as well: https://docs.k0sproject.io/stable/high-availability/#load-balancer

This link, right at the top indicates.

Control plane high availability requires a tcp load balancer, which acts as a single point of contact to access the controllers.

This is not "comprehensive" and leaves out many points about where you would install this. More important is that a single load balancer would become a single point of failure. I was attempting to implement the information at https://docs.k0sproject.io/v1.31.1+k0s.1/cplb/.

Re: "on topic", I believe this is relative and on topic, as you want to be aware of things like this in making this a CNCF project. If I am wrong, then I apologize and will leave this as my last post here.

[twz123]

I configured a test cluster using this page and yet I still have

sage@sage:~/work % ssh ubuntu@work1 sudo grep server /var/lib/k0s/kubelet.conf
    server: https://192.168.6.21:6443

This is the IP address of the ctrl1 node and not a load balanced IP. BTW, all the worker nodes have this. I don't see anywhere that lets me know how to set it up to be a load balanced IP address for the control plane. It is very possible I just don't understand or missed where it says this and how to set it.

NB: When NLLB is enabled, the actual kubeconfig being used is located at /run/k0s/nllb/kubeconfig.yaml.