Package Calico as a Helm chart

[tomdee]

See https://github.com/kubernetes/helm/blob/master/docs/charts.md

[caseydavenport]

Last I checked there was a dependency on networking in order to use helm - the tiller Pod needed networking installed in order to work, which meant installing networking wasn't possible through Helm.

This was many months ago, so may have changed since!

[caseydavenport]

Helm now allows running the tiller Pod using host networking, but we're still currently blocked on this: kubernetes/kubernetes#45717

[luckyfengyong]

The kubernetes/kubernetes#45717 has been closed due to the feature of TaintNodeByCondition in K8S 1.9. The ticket is unblocked, I think.

[caseydavenport]

@luckyfengyong thanks for the heads up!

Looks like it's an alpha feature, so would be good for us to try it out and see if it meets our needs.

[mhobotpplnet]

Here is aws helm chart for calico in case anyone is looking for it. Still needs some values restructuring, but its a first stab at it.
https://github.com/mrzobot/calico-helm-chart

[jayunit100]

I'll test this and get back to you . Thanks @mhobotpplnet .

[jayunit100]

@caseydavenport do you have a preference for where this lives ?

1. where should it live?

• I'm assuming, stable/ upstream,
• projectcalico/helm-chart
• something else?

2. where should we test it?

• assuming we have a ci pipeline somewhere.
• if someone could introduce me to who runs it, and how i could add jobs to it, that would be cool.

[secret104278]

Is there any update of this issue?
After helm 3 was released, the tiller pod was removed, so it probably will be more straight forward to deploy calico by helm chars.

[caseydavenport]

Currently, the core Calico team is not really able to be in the business of maintaining a Calico helm chart. There are a lot of different ways to configure and package Kubernetes applications and we can't maintain and test all of those ourselves, so we need to rely on others to do that packaging.

I think there are some third-party helm charts for Calico out there in the wild, though.

[derekhulley]

@caseydavenport, understood: a supported Helm chart cannot be maintained without great expense and will only work for specific conditions. A link to a minimal chart used for system testing is useful: there is one in the code and an "in the wild" link already provided in this issue. It should suffice for further development.

[invidian]

I also published Calico helm chart recently, as it is used by my project during cluster bootstrapping. It's available via https://flexkube.github.io/charts charts repository. Currently it's all defaults in there, but I'm happy to implement requested changes.

It would be great if Calico team could maintain that, as with Helm 3, it's very easy to use Helm for managing Calico deployment.

[rochaporto]

Will this be it? https://github.com/projectcalico/calico/tree/master/_includes/charts/calico

[kfox1111]

Yep, I appreciate where you're coming from. Let me add my thoughts and see if they make sense:

Yup. Thanks for the discussion. :)

Absolutely, we're working on improving that - the RBAC in that file there isn't representative of the true number of permissions the code needs. In other words - it's giving unnecessary permissions right now. We can even use Kubernetes RBAC to limit to specific resource names, which will really tightly scope the power of the operator.

👍

its auditable by the end sysadmin

I'm not sure I buy this. At the end of the day, the helm chart would be deploying some sophisticated software into the cluster that is not audit-able except by someone who has read the source code. Both Calico and Kubernetes itself are both made out of of a set of compiled controllers written in golang, just like an operator is.

Hmm... yes and no. I wasn't scoping all of Calico into my statement but I guess I did not state that. I was more referring to the packaging. If I can quickly audit the deployment scripts, I know roughly what box the software under deployment is restricted to. I can quickly look at the rendered templates and see what its going to do. With an operator, you can't see what it always will do, just what its done after the fact. So the best you can do is look at the operator's rbac rules and guess. That does relate to previous concern, which would be better when tightened. So will be less of a problem later.

it doesn't have a bad dependency stack during bootstraping. core k8s, helm install, working cluster.

I'm not sure I follow here - an operator based approach is the same set of dependencies. Core k8s, kubectl apply the operator+CR, working cluster. Am I missing something?

I guess I might have been mixing concerns a little bit here. If you were to deploy the operator using the OLM, I think you run into bad bootstrapping dependencies. If simply using helm to deploy the operator, its not a problem. Without something like the olm, I'm not sure how much helm installing the operator vs just helm installing calico directly buys you for the extra complexity.

How does the operator get updates?

As a basic level, it's as simple as bumping the image version, maybe make some manifest changes. I see helm as a lightweight way to manage upgrades of the operator itself, rather than using something like OLM (which could also be an option for those who want it, though as you said would need some special steps to work with a network operator)

Ok, yeah.

So once permissions are tightened up, I think the main trade off I see, is more obscurity on the deployment for sysadmins because its harder to see the manifests before the operator generates them vs the ability for the operator to do complex lifecycle operations on Calico to save the sysadmins work? So work for an sysadmin doing some verification vs work for a sysadmin doing more complicated upgrades.

I think the canal -> calico upgrade procedure was more complicated then just a helm upgrade, but was a one time thing and even with the few extra steps wasn't a huge deal.

Do you envision much need for more complicated upgrade procedures beyond what a simple helm upgrade could do?

Appreciate your feedback. I'll continue to chew on this a bit.

Yeah, me too. Thanks again for the discussion/thought.

[cabrinha]

Can we just get this packaged up as a helm chart? https://github.com/projectcalico/calico/tree/master/_includes/charts/calico

[caseydavenport]

@cabrinha please see my earlier comments on this - that chart is for internal website rendering purposes only.

I want to point out that those helm "charts" are not a supported API / install mechanism. We don't consider the logic in those charts to be suitable for general use and we don't test all of the permutations.

[caseydavenport]

So once permissions are tightened up, I think the main trade off I see, is more obscurity on the deployment for sysadmins because its harder to see the manifests before the operator generates them vs the ability for the operator to do complex lifecycle operations on Calico to save the sysadmins work?

@kfox1111 Yes, this is certainly part of the benefits it brings (handling complex lifecycle operations). But like you said, those scenarios are relatively rare. There are other, perhaps more subtle, benefits as well, that it brings to every release.

• Reduces the potential for misconfigurations by putting an API with validation and defaulting on top.
• Reconciles changes to ensure the cluster stays in its desired state and accidental changes are not made.
• Provides a more scalable default configuration, including auto-scaling for components like Typha.
• Provides a better default security posture by provisioning TLS certificates on connections that use them by default (e.g., calico/node to Typha).
• Ensures that what you get is what is tested and supported. I've often seen clusters fail when a user decides to use the same manifests they have been using for Calico v3.6, but bump the image version to Calico v3.14 (as an example).
• Consistent experience across all platforms (e.g., OpenShift, which doesn't allow installation of network plugins via helm).

Of course, some of these things are achievable via a helm chart. But also some of them are not.

[vhosakot]

Now that latest helm does not have tiller, does it work without CNI? If so, calico can have helm charts and can be installed by helm? Thoughts?

[kfox1111]

It works without cni.

[jotcodeofficial]

Would love an official helm chart

[vhosakot]

Any reason there are no official helm charts to install calico or calico operator?

Docs say we need to create the manifests (calico.yaml or tigera-operator.yaml) using kubectl.

Looking at the manifests, it looks like:

• calico.yaml manifest renders the calico helm charts at https://github.com/projectcalico/calico/tree/master/_includes/charts/calico but it doesn't have values.yaml

• tigera-operator.yaml manifest renders the calico operator helm charts at https://github.com/projectcalico/calico/tree/master/_includes/charts/tigera-operator but it doesn't have values.yaml

Can these manifests be converted to values.yaml so calico or calico operator can be installed using helm instead of kubectl?

[vhosakot]

It works without cni.

Thanks. I made helm charts for calico by adding values.yaml for https://github.com/projectcalico/calico/tree/master/_includes/charts/calico and was able to install it using helm install.

So, yes, helm does work without CNI. I'm using helm v3.2.4:

$ helm version
version.BuildInfo{Version:"v3.2.4", GitCommit:"0ad800ef43d3b826f31a5ad8dfbb4fe05d143688", GitTreeState:"clean", GoVersion:"go1.13.12"}

[cabrinha]

How about getting this packaged up as a helm chart? https://docs.projectcalico.org/manifests/calico-vxlan.yaml

[ReSearchITEng]

@invidian ->

as you already maintain the calico non-operator charts in your helm repo (#717 (comment)), can you add also the operator one (https://github.com/projectcalico/calico/tree/master/_includes/charts/tigera-operator ) along with a values.yaml ?

[invidian]

@ReSearchITEng I'm not actively using tigera-operator, so perhaps it would be better if someone else would maintain it.

[zamazan4ik]

Any updates? Is there any guide, how to install calico with a helm chart?

[cabrinha]

This issue has been open since 2017. Can we get a helm chart made for calico please?

[im-nonedone]

@caseydavenport so I made some changes to the helm chart and added values.yaml file and a README.md. It is after pushing it to my Github I realized values.yaml files are ignored for charts. Is there a reason why this can't be allowed? This along with the readme will help others to download and use this chart locally at least. I can create a pull request if you are fine with the approach.
https://github.com/im-nonedone/calico/tree/nsadineni/added-values-readme/_includes/charts/calico

[caseydavenport]

CC @ozdanborne and @lwr20 who are currently working on a helm3 chart, and know better than I.

[im-nonedone]

@ozdanborne @lwr20 can you guys take a look at this and see if this can be packaged as a helm chart. I changed it to helm 3, added crds folder and added readme.
https://github.com/im-nonedone/calico/tree/nsadineni/added-values-readme/_includes/charts/calico

[lwr20]

For the benefit of those watching along, the helm3 charts work has progressed to the point that you can build helm3 charts from this repo:

git checkout origin/master
make charts

and the charts tgz will be built and can be found in /bin/

This works for master and release-v3.17 branches at present.

Work is still ongoing on adding automated testing for the resulting helm bundle and on the machinery to actually add a helm charts bundle as a release artifact.

[lwr20]

Helm chart bundles are now being added to the release artifacts. e.g.
in https://github.com/projectcalico/calico/releases/tag/v3.18.1 the helm chart is tigera-operator-v3.18.1-1.tgz

[viceice]

How do i need to configure helm to fetch those bundles? Normally you you need a helm repo.

Is it planned to publish them as helm repo or do we need to create our own repo for it? 😕

[lwr20]

There's no need for a helm repo:
https://docs.projectcalico.org/master/getting-started/kubernetes/helm

But I can see that it would make finding the latest chart easier. I'll raise an issue to track that.

[lwr20]

Raised #4514

Please add comments to that to explain why this would be a good thing to have so that this work can be prioritised appropriately.

[randrusiak]

@lwr20 If I understood correctly currently there is only helm chart for tigera operator. But what about calico?
Do you consider to release this chart as well?

[lwr20]

If I understood correctly currently there is only helm chart for tigera operator. But what about calico?

tigera-operator installs Calico. (It can also be used to install Calico Enterprise, but that's not relevant here).

[randrusiak]

I've never used tiger-operator and I thought it's only for calico enterprise, but now everything is clear for me, thanks :)