Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Official Helm Chart #572

Closed
stevehipwell opened this issue Aug 12, 2020 · 56 comments · Fixed by #670
Closed

Official Helm Chart #572

stevehipwell opened this issue Aug 12, 2020 · 56 comments · Fixed by #670
Labels
kind/feature Categorizes issue or PR as related to a new feature.

Comments

@stevehipwell
Copy link
Contributor

What would you like to be added:

An official Helm Chart.

Why is this needed:

Because the stable repo is due to be retired and an official Helm chart is really useful for the community.

Prior art:

kubernetes-sigs/descheduler#298
kubernetes/autoscaler#3341

/kind feature

@k8s-ci-robot k8s-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Aug 12, 2020
@serathius
Copy link
Contributor

serathius commented Sep 24, 2020

I was a little skeptical for creating separate distribution of manifests in addition to Kustomize. Not everyone uses helm, so Kustomize made more sense as there was Metrics Server chart already maintained by community.

Looking at links you provided, I like the approach taken of separating chart owners and releases from main release. Charts linked are also pretty high quality.

I would be for adding helm charts if 2 maintainers step up to introduce and maintain the chart. Current Metrics Server maintainers do not use helm, meaning that they would not be capable to maintain high quality. We should also figure out how do we maintain both Kustomize and Helm configuration.

/cc @s-urbaniak
What's your opinion?

@olemarkus
Copy link

I am one of the maintainers of the helm/stable repo chart and would be happy to maintain the chart as part of this repo instead.

@stevehipwell
Copy link
Contributor Author

@olemarkus you might want to keep an eye on helm/chart-releaser-action#37 which would allow you to keep the chart name metrics-server but have additional information in the tag name (e.g. _metrics-server-chart-vX.Y.Z).

@scottrigby
Copy link

👍 a working demo steps for all the Helm GitHub actions is here: https://github.com/helm/charts-repo-actions-demo

@scottrigby
Copy link

I've linked this issue in the table at https://github.com/helm/charts/issues/21103.

@serathius There is also a kubernetes organization on the CNCF Artifact Hub (https://artifacthub.io), which hosts CNCF packages like Helm charts, OLM Operators, Falco Rules, and OPA Policies. Would you and/or other metrics-server maintainers like to be added to that so you can manage the metrics-server charts repo? There's not really much to do after setting it up, verifying, and marking "official". If so, lmk and I can add you, and help with whatever needed there.

@scottrigby
Copy link

@olemarkus if you are going to be helping with this – or whoever is – you can look at other previous migrations from stable, such as https://github.com/prometheus-community/helm-charts, https://github.com/grafana/helm-charts, and https://github.com/jenkinsci/helm-charts.

These examples all followed current best practices by importing git history (I've laid out example steps here: prometheus-community/helm-charts#3), and set up Helm GitHub actions for CI (chart testing), and CD (releasing chart version packages to the new chart repo).

Please let me know if you have any questions or would like some help with this.

One important note: this is time-sensitive for end users of the chart. stable and incubator chart repos will be end of life on 13 Nov 2020.

@olemarkus
Copy link

Looked easy enough. Filed #606 as a start.

The old merge request PR references are naturally broken in the history, but I see other migrations have ignored that part as well.

@serathius
Copy link
Contributor

PTAL @s-urbaniak

@olemarkus
Copy link

The one thing that is a bit unclear here is how to handle chart releases vs metrics-server releases. I guess that is what helm/chart-releaser-action#37 tries to support.

@s-urbaniak
Copy link
Contributor

I agree with the statements of @serathius. We, for instance, do not use helm charts either, but rather rely on jsonnet for managing our manifests. As stated there are also kustomize, etc, hence I am very sympathizing with the following statement:

I would be for adding helm charts if 2 maintainers step up to introduce and maintain the chart. Current Metrics Server maintainers do not use helm, meaning that they would not be capable to maintain high quality. We should also figure out how do we maintain both Kustomize and Helm configuration.

An experience point from maintaining prometheus-operator (https://github.com/helm/charts/tree/master/stable/prometheus-operator). The helm charts are maintained there, in a separate repository, by separate maintainers, treating prometheus-operator releases as "upstream". I suggest a similar model here.

@serathius
Copy link
Contributor

@s-urbaniak I think the problem is that helm/charts repository is being deprecated and those charts don't have place to live. Looks like prometheus-operator was also moved.

@scottrigby
Copy link

@serathius I think @s-urbaniak 's point still stands. The https://github.com/prometheus-community/helm-charts/ chart repo is in a separate git repo. One of the side benefits of that is that the github releases aren't listed along with the upstream prometheus projects each chart installs.

Though I understand kubernetes-sigs/metrics-server is a bit different, because a separate charts repo may be confusing in the kubernetes-sigs github org. An example of k8s apps maintaining a chart repo in the same git repo as the upstream project is https://github.com/kubernetes/ingress-nginx/

@bryankaraffa
Copy link

I think @scottrigby summed up the two different models that other projects are going with. As a user I have no preference, as long as the chart is published in a proper helm chart repo I can add or declare.

Wherever the chart ends up living, looks like it's easy to "to turn a GitHub project into a self-hosted Helm chart repo":
https://github.com/helm/chart-releaser-action
https://github.com/kubernetes/ingress-nginx/blob/master/.github/workflows/helm.yaml

@stevehipwell
Copy link
Contributor Author

@olemarkus I've created a PR kubernetes-sigs/descheduler#436 to update the descheduler chart to use the new tag naming template allowing it to live in the same repo as the code.

@baderbuddy
Copy link

@olemarkus it look like you were the only one that's offered to maintain this chart. I just wanted to offer that if you need any assistance I'm pretty experienced with Helm so I could help too.

@chadlwilson
Copy link

chadlwilson commented Dec 4, 2020

We migrated to https://github.com/bitnami/charts/tree/master/bitnami/metrics-server (which gets regular upstream docker image rebuilds as well) since it was a bit worrying that this didn't seem to be getting the attention it needed (i.e the earlier PR merged/reviewed).

I submitted a PR to remove the main delta I saw from this chart to that one (which was promptly merged); but interested in any thoughts others in the community might have of suitability of one chart over the other.

@olemarkus
Copy link

Due to the time this has taken, my employer plan on using the built-in addon in kops 1.19 instead. I may still contribute to an official chart, but I don't think I can commit to being one of two main contributors.

@walkafwalka
Copy link

Our company avoids implementing services without official Helm charts. It is a sign of product maturity and we do not have the bandwidth or desire to maintain a chart that is compatibility with version releases. I will probably also look into bitnami's chart especially since it supports 0.4.0 but an official Helm chart is still preferred.

@stefansedich
Copy link

Due to the time this has taken, my employer plan on using the built-in addon in kops 1.19 instead. I may still contribute to an official chart, but I don't think I can commit to being one of two main contributors.

Happy to throw my name in the ring to help with the chart @olemarkus @baderbuddy if that eases the burden to get this one out the door.

@stevehipwell
Copy link
Contributor Author

I'm happy to put myself forwards to help with this. I've already been involved with the descheduler chart directly and have been part of the discussions around the cluster-autoscaler chart.

@scottrigby
Copy link

Awesome. PS there's also a kubernetes-sigs org now at artifacthub.io - if maintainers in this org want to access that to add this chart (or @stevehipwell, also the descrheduler chart), lmk.

@stevehipwell
Copy link
Contributor Author

@scottrigby the descheduler chart repo is currently listed on artifacthub.io. My understanding is that the organisation would take ownership of the repo, do you have any more info on how this would work?

@scottrigby
Copy link

@scottrigby the descheduler chart repo is currently listed on artifacthub.io. My understanding is that the organisation would take ownership of the repo, do you have any more info on how this would work?

@stevehipwell Yes. First, you must be a member of the org. By default (unless fine grained permissions are configured for the org using an OPA policy), members can add other members. I can help add whoever is responsible for this repo, and from that point on they can help to add other members as needed too. There is an expectation that trusted people will only add other trusted people (this is where configuring fine-grained permissions can be helpful though, to minimize accidental errors, especially for very large orgs. By default permissions are just flat).

Once a user is part of the org they want to transfer a repo to, they can create an Ownership Claim (see that section in the Artifact Hub repositories guide).

The Control Panel has inline documentation on how to use the repo transfer functionality (allows to transfer a repo between personal and/or any org you're a member of).

Hope that helps! Please lmk if I can help more. Also there is an #artifact-hub channel in CNCF Slack, in case I'm unavailable at any point.

@cpanato
Copy link
Member

cpanato commented Dec 18, 2020

We can host the helm chart similar to the dashboard is doing https://github.com/kubernetes/dashboard/tree/master/aio/deploy/helm-chart

what do you think?

@scottrigby
Copy link

scottrigby commented Dec 19, 2020

@cpanato Can you summarize the differences between https://github.com/helm/charts-repo-actions-demo and how kubernetes/dashboard is hosting a helm repo? There have been a number of kubernetes projects now hosing their own helm chart, and each do it a little differently, which is ok. But it may be helpful to arrive at and recommend a common pattern for other k8s projects who want to host their own chart repo. If we describe the differences in the existing examples it may help us get closer to that. What do you think?

@cpanato
Copy link
Member

cpanato commented Dec 20, 2020

@scottrigby makes total sense, lets first check if the way we are trying for will work, then we can suggest a recommended way because I think that pattern will be easier compared with the one I wrote above.

@scottrigby
Copy link

scottrigby commented Dec 20, 2020

@cpanato perfect. @torstenwalter compiled a good step-by-step guide based on work I and he had done on this for various community chart repos: https://github.com/torstenwalter/helm-chart-hosting. This includes how to port over git history using existing tools, and also package history using a tool I wrote (I don't think it includes steps for charts that have already been relocated without git history, but it's very possible to bring that back in too - here is one example I worked on: goharbor/harbor-helm#48), as well as setting up CI/CD with Helm GitHub actions.

Here's a recent example PR I made for a k8s project: kubernetes/kube-state-metrics#1237, if you want to look at how the gh actions workflow files are put together (there were some follow-up PRs as well so check master when looking at the workflows). I know some other k8s projects have done it a bit differently.

I also want to second what @stevehipwell said #572 (comment) about the name template: CR_RELEASE_NAME_TEMPLATE is available as of helm/chart-releaser-action v1.1.0.

@cpanato
Copy link
Member

cpanato commented Jan 12, 2021

@stevehipwell I think that is ok from my side, but I'm not a maintainer

however as soon you open that I will do the infra side of things :)

@stevehipwell
Copy link
Contributor Author

@cpanato are we looking at the same workflow (GitHub actions) that's used for descheduler and cluster-autoscaler (I've worked on both) or another one?

@stevehipwell
Copy link
Contributor Author

Are there any maintainers who would like to comment on this?

@jdambly
Copy link

jdambly commented Jan 19, 2021

@olemarkus

@olemarkus my company also requires that all add-ons be deployed with helm, so I am happy to assist as well. I just need to get plugged into the community. please let me know what I can do to help.

@stevehipwell
Copy link
Contributor Author

@cpanato due to the lack of comments I'll go ahead and create a PR for the chart and then see where we go from there.

@Sliuza
Copy link

Sliuza commented Jan 26, 2021

@stevehipwell , did you have any progress? I'm also interested on this and I can help with the PR.

@stevehipwell
Copy link
Contributor Author

I'm planning on creating the PR this week.

@stevehipwell
Copy link
Contributor Author

I'm waiting on feedback for PR #670.

@pierluigilenoci
Copy link
Contributor

@stevehipwell another two months and no progress. 😢

@stevehipwell
Copy link
Contributor Author

@pierluigilenoci I know. I'm resisting putting it in my own chart repo as I'd much rather it was alongside the code and maintained as a single unit.

@stevehipwell
Copy link
Contributor Author

I've released my own version of the chart which I will deprecate when #670 is merged.

https://artifacthub.io/packages/helm/stevehipwell/metrics-server
https://github.com/stevehipwell/helm-charts/tree/master/charts/metrics-server

@liangyuanpeng
Copy link

The helm of the Metrics Server is necessary and i look forward to Official Helm Chart can continue to go.

@pierluigilenoci
Copy link
Contributor

Soon this issue turns one year old.
@stevehipwell are you planning any celebrations?

@elcomtik
Copy link

elcomtik commented Jun 28, 2021

My opinion is that this issue will not be resolved. I looks that you can develop open-source code, but not merge it here, unless you have sponsor which backs your maintainership.

@h0jeZvgoxFepBQ2C
Copy link

Could someone tell me what is now the recommended way to install the metrics server via helm?
Via bitnami or via the helm repo mentioned in this repository?

@onedr0p
Copy link

onedr0p commented Sep 23, 2021

Could someone tell me what is now the recommended way to install the metrics server via helm?
Via bitnami or via the helm repo mentioned in this repository?

Whatever you want, for me I've decided to use the official chart here.

@h0jeZvgoxFepBQ2C
Copy link

But are there any differences?
It's not really clear to me if there are any differences?

And do you think I can install the offical helm chart, even when I installed it before via the kubectl apply -... command? thank you!

@pierluigilenoci
Copy link
Contributor

Enjoy https://github.com/kubernetes-sigs/metrics-server/tree/master/charts/metrics-server

@stevehipwell
Copy link
Contributor Author

@h0jeZvgoxFepBQ2C
Copy link

h0jeZvgoxFepBQ2C commented Sep 23, 2021

Yeah I know there is the Readme, but this doesn't answer my question? 😅

I did ask what the differences are between the binami helm chart and the kubernetes-sigs helm chart?

And also if it's possible to update an installation via helm, which was installed initially via kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml? :)

@stevehipwell
Copy link
Contributor Author

@h0jeZvgoxFepBQ2C you'd have to compare the Helm output of the two charts to see the full set of differences. I'm not an expert on Bitnami, but the official chart will give you the official image and configuration, Bitnami uses a custom image and has other differences.

RE manifest to chart - the official chart is based on the official manifests but customisable. I'm sure it would be possible to either deploy the chart over the manifest or customise the manifest so the chart could capture the resources. However, my question would be why go to the effort? MS is stateless, so deleting the manifest and installing the Helm chart would be comparable in down time to an upgrade of the manifest.

@tbondarchuk
Copy link

@h0jeZvgoxFepBQ2C I've simply run helm template on both charts and compared differences. From what I remember, besides bitnami's custom image and chart labels, main difference were rbac permissions and args. The only thing that matters to me was insecureKubeletTLS arg so I've just added it to official chart custom values and reinstalled. Everything seems to work just fine since then.

@h0jeZvgoxFepBQ2C
Copy link

Thanks so much @aliusmiles !

@SohamChakraborty
Copy link

@pierluigilenoci @stevehipwell Sorry for bothering, is there any chance of getting a helm chart for metrics-server 0.4? If I am not wrong, the current chart is for version 0.5 and there is no chart for 0.4. The deprecated stable repo also doesn't have a 0.4 chart. So I am wondering what are my installation options for EKS 1.18? I may try with version 0.5 as compatibility matrix shows, but I wanted to go to 0.4 from 0.3 before moving to 0.5.

Any suggestions? Thanks!

@stevehipwell
Copy link
Contributor Author

@SohamChakraborty we're running the current v0.5 version on EKS v1.18. Your other options are; my personal repo, which is where the official chart came from, has v0.4 versions; or just change the image tag on the official chart (I can't guarantee this works, but it should).

@pierluigilenoci
Copy link
Contributor

@SohamChakraborty version 0.5 supports k8s 1.18+, I recommend you try it and see if it works.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature.
Projects
None yet
Development

Successfully merging a pull request may close this issue.