Automate Helm Upgrades based on semver expressions

[srueg]

Would it be possible to instead of watching images for updates, watching helm charts for new versions? The flux.weave.works/automated: "true" annotation could be set on a HelmRelease CR.

[squaremo]

It certainly is possible, and desirable. The difficulties are more architectural than technical: at present, fluxd only "knows" about manifests and images, and adding Helm charts to that seems headed in the direction of being monolithic.

I would like to move the image database and automation into its own process/container/image; if that can be done, scanning and updating chart versions is a close analogue and could be done in the similar way. But I'm not set against adding this into fluxd and moving it out later -- just conflicted about it :-/

(BTW you already can use the annotation flux/weave.works/automated in a HelmRelease -- it controls image updates. We'd have to use another annotation, or have some other way to distinguish resources for which you want the chart version automated.)

[srueg]

Thanks for the fast reply! Cool, so this is something you're planning / working on 👍 Maybe I find the time to code together a first draft so we have something to reason about.
Merry x-mas

[k]

I was just thinking about this and turns out there was already an issue about this!

One reason I think upgrading helm charts instead of images when using helm for each application is because:

1. allows developers to install individual helm charts at specific versions themselves if needed
2. Still keeps the deployment manifests in the same location as the application code
3. Can fully automate pushes to a dev cluster

The problem with using git-based helm chart updating is you either have to put the charts in one repository or have a bunch of separate repositories that flux is pulling from, which seems like an antipattern in the gitops paradigm. Alternatively one could use a monorepo, but I think that gets messy for other reasons.

[davidkarlsen]

Looking for the same.
My suggestion would be:

• no version specified -> use the latest available version
• allow specifying a wildcard: 1.1.*, or 1.*
• specify an exact version - as today

[squaremo]

One reason I think upgrading helm charts instead of images when using helm for each application is because [...]

I fully expect people to want to update the chart version as well as the images used with the chart. Chart versions will tend to change when new images are not backward-compatible; but also on their own terms, e.g., if there's some bit of extra operational config you can supply to the chart. You don't always need or want to bump the chart version with each new image.

[squaremo]

My suggestion would be:

• no version specified -> use the latest available version
• allow specifying a wildcard: 1.1.*, or 1.*
• specify an exact version - as today

I think the above is implicitly suggesting that these values can be used in the version field of a HelmRelease -- I don't think that's a good idea though, since we generally want to record the state as precisely as we can, and put the policy for automation elsewhere. Otherwise there'd be no record in git of the change when the chart is upgraded to a new version.

[stephenmoloney]

Given that the values.yaml manifests typically change from one version of a helm chart to the next, is it even plausible that this is something that can be automated ?

[k]

Couldn't you say the same thing about docker images and env variables

[stephenmoloney]

Couldn't you say the same thing about docker images and env variables

Fair point, it's up to developers I suppose to decide on the appropriateness of the automation.

[k]

I think the above is implicitly suggesting that these values can be used in the version field of a HelmRelease -- I don't think that's a good idea though, since we generally want to record the state as precisely as we can, and put the policy for automation elsewhere. Otherwise there'd be no record in git of the change when the chart is upgraded to a new version

@squaremo

This can still be done if the version schema is in an annotation and then the current version is in the version field of the helm release crd

[squaremo]

This can still be done if the version schema is in an annotation and then the current version is in the version field of the helm release crd

Yes, for sure. I'd prefer it acted like the image update automation; to wit, you say which versions are acceptable in an annotation, and the automation updates the version field accordingly, committing the change to git.

[captncraig]

Should helm chart polling be done by the helm operator or by fluxd? Fluxd already has a very similar paradigm with config for the git repo and rate limiting config and all that. But helm operator interacts with chart repos and fluxd doesn't. It makes sense to me for helm-operator to watch for changes.

[captncraig]

Nvm, it seems fluxd is definitely the right place, if this autoamation will be committing back to the master repo.

[squaremo]

It makes sense to me for helm-operator to watch for changes.

vs

fluxd is definitely the right place, if this autoamation will be committing back to the master repo

Yep -- that's the quandary! The "clean" way to do this would be the automation (and registry scanning) bits to be separated from the synchronising bits -- though there are some bits that are tricky to disentangle, like access to the git repo -- and of course that would mean more operational burden (at the least, more config).

[andrejpetras]

Hi @stefanprodan, @hiddeco, @squaremo

we want to use the fluxcd in the large project (multiple teams, multiple services) but the release process should be based on helm chart version and not the docker image version.

Are there any news or plans for this issue?

[kalebwalton]

Agree - our releasable artifact is a combo image + helm chart. The Helm chart will always have the latest image reference.

[fredgate]

Idem. I want to use fluxcd and helm operator, but I want that my config repo be updated when a new chart version is available in a helm repository (I do not want to poll a docker registry).

[hiddeco]

We are still actively tracking interest in this, and are aware of the growing interest, but are kept back by the fact that it does not belong in the Flux code base while the operator itself is not the best candidate either.

There are plans to move the image update feature from Flux to a dedicated service, once this has happened the path for this enhancement should be more clear.

[fredgate]

I agree that it is not the responsabilty of helm operator to update git repository as its role is just to deploy charts referenced by release defined in CRD in cluster.
It is more the job of flux to auto update the git repository. But effectively it must be enhanced to update different information (version of images or of charts) in different scenarios (new artifact in the docker registry, or in the helm registry). Extract update of images in a dedicated service should effectivly allow to add another service for update of charts.

[joberdick]

@fredgate @hiddeco To clarify, what is the current thought of the operator having the ability to watch Helm repo charts for updates directly?

I agree on not necessarily having the operator update git. Right now, helm operator needs flux to get updates from helm repos. Flux is a nice tool, but way more complicated than necessary if you simply need to install updates from pure helm charts.

[fredgate]

I am not interested in the fact that the operator could update a release when a new chart is available in a Helm repository. Because for me the right approach is to have a CD pipeline based on GitOps. And Flux is perfect for that.
However Flux consider to extract in an external service, the monitoring of new images in the Docker registry and updating of the Git repository accordingly.
I think we could have a similar external service to monitor new charts in a Helm repository. This service could be configured to allow two types of updates: either by updating the manifest of the HelmRelease in the Git repository, or by modifying the HelmRelease directly in the Kubernetes cluster. In my opinion the first is the most important. But the second could then be added quite easily.

[jbalonso]

Would it make sense for helm-operator to report potential new chart versions in the crd's status field? I vaguely suggest that the operator filter down according to version matches (to avoid spamming the status). Then, fluxcd can review the data but maintain responsibility for applying a change policy.

[stefanprodan]

The helm-controller (part of the GitOps Toolkit) is capable of tracking semver releases and automatically upgrade a HelmRelease based on a semver range, here is an example: https://toolkit.fluxcd.io/guides/sealed-secrets/

[squaremo]

The helm-controller (part of the GitOps Toolkit) is capable of tracking semver releases and automatically upgrade a HelmRelease based on a semver range

The original comment asked for a chart version analogue to image updates -- in GOTK, this would be a controller that monitors helm charts and commits changes to HelmRelease YAMLs in the git repo.

[joberdick]

The helm-controller (part of the GitOps Toolkit) is capable of tracking semver releases and automatically upgrade a HelmRelease based on a semver range

The original comment asked for a chart version analogue to image updates -- in GOTK, this would be a controller that monitors helm charts and commits changes to HelmRelease YAMLs in the git repo.

I must have missed that. I would not be asking for that. All I was after is helm operator to be able to update base on new chart version. Seems like v2 operator might have this solved.

[kingdonb]

Sorry if your issue remains unresolved. The Helm Operator is in maintenance mode, we recommend everybody upgrades to Flux v2 and Helm Controller.

A new release of Helm Operator is out this week, 1.4.4.

We will continue to support Helm Operator in maintenance mode for an indefinite period of time, and eventually archive this repository.

Please be aware that Flux v2 has a vibrant and active developer community who are actively working through minor releases and delivering new features on the way to General Availability for Flux v2.

In the mean time, this repo will still be monitored, but support is basically limited to migration issues only. I will have to close many issues today without reading them all in detail because of time constraints. If your issue is very important, you are welcome to reopen it, but due to staleness of all issues at this point a new report is more likely to be in order. Please open another issue if you have unresolved problems that prevent your migration in the appropriate Flux v2 repo.

Helm Operator releases will continue as possible for a limited time, as a courtesy for those who still cannot migrate yet, but these are strongly not recommended for ongoing production use as our strict adherence to semver backward compatibility guarantees limit many dependencies and we can only upgrade them so far without breaking compatibility. So there are likely known CVEs that cannot be resolved.

We recommend upgrading to Flux v2 which is actively maintained ASAP.

I am going to go ahead and close every issue at once today,
Thanks for participating in Helm Operator and Flux! 💚 💙