Add Cilium as a Overlay Network

[mmack]

What feature would you like to be added?
Cilium as a overlay

What are use cases of the feature?
Cilium has lot of benefits over a calico / kube-proxy setup, described in detail here: https://cilium.readthedocs.io/en/stable/intro/.
For me the 3 outstanding features are:

• FQDN support for networkpolicies
• BPF Support
• Removal of kube-proxy and iptables
• Much better monitoring / insights

[kubermatic-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[xmudrii]

/remove-lifecycle stale

[kubermatic-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[kron4eg]

There was added "external" CNI support added, so any CNI plugin can be installed

more info: #862

[shibumi]

Sad, that cilium isn't included in the plugins that can be directly integrated via kubeone.

[kron4eg]

@shibumi we will accept the PR with cilium addon ;)

[shibumi]

@kron4eg ah ok. So you are open for a possible cilium addon? Will have a look on this maybe. The Canal part doesn't look that complicated. If I read this correctly you are just composing the canal deployment directly via the Kubernetes API, right? So everything what I would need to do is:

• Reading the YAML files for bootstrapping Cilium
• Writing the necessary Go/Kubernetes-Client Code for the deployment
• Make things dynamic that we can pass variables to the deployment.

Is this correct?

[kron4eg]

@shibumi while yes we are open, I meant "addons" mechanism.

Existing example would be Calico VXLAN addon. So no Go code is necessary.

[ag1989]

@shibumi would be very nice to see this feature in KubeOne :)

[shibumi]

@kron4eg this sounds even easier than the Go code.

[kron4eg]

OK then I'll reopen this back.

[cedi]

@shibumi can you ping me in the CNCF Slack? I'm also very interested in Cilium in KubeOne/Kubermatic.
Maybe we can do this together

[xmudrii]

/remove-lifecycle stale

[shibumi]

@cedi sure, it shouldn't be that complicated. It's really just adding the cilium YAML to the addon directory + adding some Go templating and tests I guess. Or am I missing something?

[cedi]

I'll demo implemented cilium in a fork of KubeOne: https://github.com/cedi/kubeone/tree/add_cilium_cni
It's a work in progress but hints/tips, recommendations, improvements, etc. are very welcome.

@kron4eg I know you suggested to use the yaml-way in combination with the cni.external, however I currently have a rather boring weekend with nothing else to do, so I wen't all in and implemented cilium directly into KubeOne so I can use cni.cilium natively :-) Was a kinda fun exercise :)

[kron4eg]

The reason why I was suggesting addon in form of YAML is: we plan to move even more in-go-manifests to YAML addons. We want this to avoid maintaining this in Go form, and to give people opportunity to customize stuff if they are willing to do so.

[kron4eg]

With release of Go 1.16 and its new io.Fs interface and emed package we will ship some "default addons" right with binary itself, of course with maintaining ability to customize them.

[kron4eg]

Our candidate packages "to-move-to-yaml" list for now is:

• canal
• weave
• externalccm
• metricsserver
• machinecontroller

Maybe we will change something in this list, but most likely not. Anyway, the more code is expressed in form of addons the better — since it gives users opportunity to customize monifests (e.g. upgrade components without waiting for the next kubeone upgrade)

[shibumi]

@kron4eg would have been nice to read about such plans in a roadmap :)

[cedi]

We want this to avoid maintaining this in Go form, and to give people opportunity to customize stuff if they are willing to do so.

Yeah, I totally get your point. I mean I converted the YAML to the right Go-Structs. It's really PITA. So I'll get your point and fully agree.

As I said before: I have no hard feelings regarding the Go implementation. It was a nice proof of concept for me to get a bit more into the internals of KubeOne and I had some nice learnings. So don't worry :)

With release of Go 1.16 and its new io.Fs interface and emed package we will ship some "default addons" right with binary itself, of course with maintaining ability to customize them.

This sounds great!
Do you have a roadmap for that or could you point me to the right issues regarding that implementation.
I consider helping you on that a little in my spare time.

[kron4eg]

This want-to-move of internal go package to external YAML addons is just a sporadic idea inspired by the Go 1.16 release notes :D There is no official plan, and even an issue to start implementing this.

[kron4eg]

here's #1254 umbrella issue for this addons migration.