Allow flux to watch a git tag

[antonosmond]

Describe the feature
Flux currently allows you to watch a git branch and a path within a repo but you can't watch a git tag, as far as I can tell.

It'd be really useful if flux could watch a tag in git.

Before I explain the problem I'm trying to solve, it's worth noting that we're using flux for applying static manifests only and we're not using helm/tiller or any of the docker image watching functionality.

Currently, we're adopting git flow and have release branches with SemVer for versioning which are eventually merged to master and tagged.
Unfortunately, we also have a requirement to support multiple "active" versions of our software releases across multiple k8s clusters, for example:

cluster-a: my-app/v1.2.0
cluster-b: my-app/v1.1.3

This means we can't configure flux to watch the master branch of the my-app repo as HEAD may not be the version we want. Instead we have to watch the respective release branches. The problem here is that to upgrade from one version to another it requires re-configuring flux i.e. changing the branch that it watches.

If flux could watch a tag, it'd work nicely with git flow i.e. once we've merged our release branch to master and tagged it with the relevant version, we could create an additional tag for flux to watch and move the tag as needed. This negates the need to reconfigure flux. As an example, assume the below are tags in the master branch for the my-app repo:

c17f64b4: v1.2.0, flux-cluster-a
73f2c51d: v1.1.3, flux-cluster-b
7cea7fbd: v1.1.2

Given the above, if we wanted to upgrade the version of my-app in cluster-b, we'd just "move" the flux-cluster-b tag to be aligned with the v1.2.0 tag.

This would enable us to upgrade/rollback releases with ease simply by moving the tags around in git and we wouldn't have to reconfigure flux each time.

What would the new user story look like?
How would the new interaction with Flux look like? E.g.
What are the prerequisites for this?

1. User starts up Flux providing the --git-tag flag

Expected behavior

1. Flux checks out revision that --git-tag points to and applies the manifests
2. Git tag is moved to point at a new revision
3. When flux next polls the repo, it detects the revision to which the tag points has changed, checks it out and kubectl applies the changes

If people agree that this functionality could be useful, I'd be happy to start working on it and submit a PR.

[Perdjesk]

I have been looking at the support for tag triggered fluxcd but with slightly different use case than yours.

Based on https://github.com/weaveworks/flux-kustomize-example how to handle the code in the repository using a trunk-based git workflow with SemVer releases instead of triggering a sync at each commit in refs/heads/master (using --git-branch). The end-goal would be to be able with the above repository and fluxcd instances per environment (using --git-path) to have different SemVer releases in each environment. This would help to validate and test changes done in the shared basemanifests in a stage environment before applying them in production. At the moment with the mentioned repository and two running fluxcd instance per environment both environments will be update at the same time when a new commit in master branch modifying files for basemanifests.
In the case of a git tag trigger support (taking your proposed --git-tag and adding support of wildcard) it could be improved by having for example:

• Staging fluxcd instance with --git-path=staging and --git-branch=master
• Production fluxcd instance with --git-path=production and --git-tag=v2.*

Thus having the staging environment tracking the tip of master branch and being updated at each pull-request merged to it and production limited to SemVer releases (tags) of a specific major.
In that example it would require to update the fluxcd instance configuration before deploying a new major in production. Which in a trunk-based development would be welcomed to be done via pull-request in the repository which configures fluxcd itself. It would allow as well to be more restrictive on the selection of the git tag if necessary by using an explicit tag (e.g --git-tag=v2.1.3).

Side point to your proposal with moving tags which does not change the validity of the proposal.
The idea of moving tags has the benefits of flexibility and not to have to change the configuration of the fluxcd instances like you mentioned. However its lacks proper workflow for review before triggering a deployment. To my knowledge there is no tool providing a review workflow when creating/writing tags. Thus it is not possible to "protect" or peer-review deployment into production which might wanted for some use cases. In the end this choice should be done by the user and they should be able to chose whether they want this flexibility or a reviewable workflow. This point could be described as part of documentation.

Other relevant discussion:
#1071

Reference to code that only looks for refs/heads:

flux/pkg/git/repo.go

Line 343 in 07b3d3d

ok, err := refExists(ctx, dir, "refs/heads/"+r.branch)

Some other tool have one parameter for triggering on git tags/branches with some fallback mechanism to switch between refs/heads and refs/tags (see: https://jenkins.io/doc/pipeline/steps/atlassian-bitbucket-server-integration/). This is cumbersome and the approach of two flags per type of git references seems cleaner or have common one only accepting explicit git references.

[antonosmond]

@Perdjesk I totally agree with you in terms of a poor workflow when moving tags and a lack of peer-review. Essentially moving a tag is a force push which I would strongly advise anyone against! But like you say, it's down to the end user how they use any functionality we're proposing and that's not really the point here. It's more about whether the ability to watch git tags would be useful to some people, regardless of their deployment strategy. It sounds like it'd be helpful for any situation where you want to target multiple versions/points-in-time on a single branch which I think is what we're both talking about.
That being said, I guess it'd make sense to support any ref, not just tags?

[linuxbsdfreak]

@antonosmond . This is what i would also wish. However you had mentioned

User starts up Flux providing the --git-tag flag

Would it not be better that if flux via a K8s custom resource provide a tag release feature. Which can be manager via Gitops. The startup of flux providing a flag sounds very manual everytime a tag has to be rolled out to a K8s cluster.

[antonosmond]

@linuxbsdfreak in my use case I was thinking about moving tags which I appreciate is not a great approach but it'd work for my use case. Given I had the following commits/tags on a branch:

SHA      TAGS
5b722484 v2.0.0
b5798f18 v1.1.1
968c8164 v1.1.0,flux
a87b7edb v1.0.0

Rather than telling flux to watch v1.1.0, I'd tell flux to watch the flux tag for example, and then if I wanted to upgrade an app on my cluster to v2.0.0, I'd move the flux tag to align it with the v2.0.0 tag. In this example, I wouldn't need to reconfigure flux when I want to upgrade my release. Instead I'd just move the flux tag along.

[linuxbsdfreak]

@antonosmond . Sounds reasonable. How would it work to promote the tags to different cluster envs? At the moment i have configured flux to pull its config from a Branch. Give the below scenario you mentioned.

c17f64b4: v1.2.0, flux-cluster-a
73f2c51d: v1.1.3, flux-cluster-b
7cea7fbd: v1.1.2

[antonosmond]

@linuxbsdfreak we actually run a flux per env on a given cluster so we have we have a flux namespace with a bunch of flux deployments e.g.
flux-dev
flux-test
flux-staging
That means we can have tags per cluster/env e.g.

c17f64b4: v1.2.0, flux-cluster-a-dev, flux-cluster-b-dev
73f2c51d: v1.1.3, flux-cluster-a-test, flux-cluster-a-staging
7cea7fbd: v1.1.2, flux-cluster-b-test,
48e20af6 v1.0.0, flux-cluster-b-staging

We'd handle moving the tags in our CI process.

[linuxbsdfreak]

@antonosmond . Thx for the info. My setup as a follows

1 cluster per env which is mapped to a git branch.

for eg. K8s clusters (poc, dev, stg) and flux is configured to pull for the relevant branches in git. It would be be nice to have flux handle the scenario mentioned above along with what i have for doing a tagged release along with a rollback scenario.

[AustinSmart]

@antonosmond This would be great to have. I have the same use case as you and I think your solution would solve things nicely.

[casibbald]

@antonosmond I really would like to see this functionality implemented.
So +1 from me

[marsavela]

Even though following a tag is something I could find useful, I'd like Flux to be able to detect when a tag has been made and same for releases. In this way I could have:

Dev cluster: latest and greatest.
Stage cluster: updated when I create a new tag.
Prod cluster: updated when I create a new release.

Is any of this already possible? I guess this is also manageable with branches though.

[2opremio]

This means we can't configure flux to watch the master branch of the my-app repo as HEAD may not be the version we want. Instead we have to watch the respective release branches. The problem here is that to upgrade from one version to another it requires re-configuring flux i.e. changing the branch that it watches.

@antonosmond would it be enough to:

1. Decouple Flux's configuration from Flux itself using a e.g. ConfigMap
2. Have Flux automatically reload when the configuration changes

(1) is already in progress (see #2544 , Christmas vacation stalled it though)

You would still need to update Flux's configuration though.

[linuxbsdfreak]

@2opremio Any progress on this issue? It would be nice to have flux watch git tags and do deployments. In case of issues a rollback of the tag would suffice.

[casibbald]

Desperately need this as well

…

On Tue, 11 Feb 2020 at 14:00, linuxbsdfreak ***@***.***> wrote: @2opremio <https://github.com/2opremio> Any progress on this issue? It would be nice to have flux watch git tags and do deployments. In case of issues a rollback of the tag would suffice. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2568?email_source=notifications&email_token=AAA6C32TBB5EHAU2QYIX3R3RCKVRLA5CNFSM4JHNTEWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELMQHXA#issuecomment-584647644>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA6C35M6YRI6OIQFWOGZCLRCKVRLANCNFSM4JHNTEWA> .

[antonosmond]

I no longer have a need for this as we completely changed our workflow but I do still think it'd be a useful feature for many people inc. me in the future.
Unfortunately, I don't have the time to work on this now but would be happy to help with reviews etc if somebody else decides to pick this up.

[cdenneen]

I no longer have a need for this as we completely changed our workflow but I do still think it'd be a useful feature for many people inc. me in the future.
Unfortunately, I don't have the time to work on this now but would be happy to help with reviews etc if somebody else decides to pick this up.

Any new workflow insights that might benefit?

[linuxbsdfreak]

@stefanprodan @hiddeco . Could you provide me some insights/thoughts regarding this feature to be included in flux? I would like to have flux have a tagged way of releasing configuration. FYI.

My setup as a follows

1 cluster per env which is mapped to a git branch.

for eg. K8s clusters (poc, dev, stg) and flux is configured to pull for the relevant branches in git. It would be be nice to have flux handle tagged release for a branch along with a rollback scenario.

[casibbald]

This is functionally I am chasing as well.

…

On Wed, 4 Mar 2020 at 09:19, linuxbsdfreak ***@***.***> wrote: @stefanprodan <https://github.com/stefanprodan> @hiddeco <https://github.com/hiddeco> . Could you provide me some insights/thoughts regarding this feature to be included in flux? I would like to have flux have a tagged way of releasing configuration. FYI. My setup as a follows 1 cluster per env which is mapped to a git branch. for eg. K8s clusters (poc, dev, stg) and flux is configured to pull for the relevant branches in git. It would be be nice to have flux handle tagged release for a branch along with a rollback scenario. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2568?email_source=notifications&email_token=AAA6C37L5ZGAJXNMNXCVNWTRFYMK5A5CNFSM4JHNTEWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENW7URY#issuecomment-594410055>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA6C36IP5MKT7KN3M2P4BLRFYMK5ANCNFSM4JHNTEWA> .

[mikesir87]

I have a desire to also watch tags, but in a slightly different use case than the original author.

I'd like to use two flux deployments for different environments (even though they're most likely in the same cluster). One is watching master to serve as staging/pre-prod. The other is watching for tags to serve prod. I'd like to follow a semver approach. So, would be great to say --git-tag=1.* and then it would deploy any newly 1.* tags.

[Perdjesk]

I have a desire to also watch tags, but in a slightly different use case than the original author.

I'd like to use two flux deployments for different environments (even though they're most likely in the same cluster). One is watching master to serve as staging/pre-prod. The other is watching for tags to serve prod. I'd like to follow a semver approach. So, would be great to say --git-tag=1.* and then it would deploy any newly 1.* tags.

This pattern is similar if not exactly the same as described in #2568 (comment)

In your use case how do you plan to deploy/configure and follows GitOps pattern for the fluxcd deployment per environment watching different tags? Would the fluxcd deployment made from git repository? How fluxcd would be deployed itself?

[stefanprodan]

Would the fluxcd deployment made from git repository? How fluxcd would be deployed itself?

You could deploy Flux from CI with https://github.com/fluxcd/fluxctl-action and by using git over HTTPS

[cdenneen]

Would the fluxcd deployment made from git repository? How fluxcd would be deployed itself?

You could deploy Flux from CI with https://github.com/fluxcd/fluxctl-action and by using git over HTTPS

Now I need to translate this for GitLab-CI 🤣

[mikesir87]

This pattern is similar if not exactly the same as described in #2568

It is the same, yes. Just not the same as the original request. Should have linked to that comment as well, so my bad there.

In your use case how do you plan to deploy/configure and follows GitOps pattern for the fluxcd deployment per environment watching different tags?

Preferably, I’d only have one deployment for prod that is watching for all new git tags, just as it could be configured to watch for new image tags from a repo. I would configure it watch for any new 1.* tags and as they are created in the git repo, the fluxcd deployment would see it on its next sync.

Now I need to translate this for GitLab-CI

Haha... we’re in the same boat being a GitLab shop ourselves.

[srknc]

I believe it would be a nice feature, here is the use case;

• There are more than 5+ clusters managed by gitops repo, let's say test1, test2, production1, production2, production3 etc
• I would like to deploy a new deployment, only to one environment, this flux will listen let's say develop branch, others master.
• After testing etc, because of business requirement, still I have to manage deployments manually, for example production1 Wednesday, production2 at 2pm, production3 when business inform us to do so.

So, I would like to make master branch ready-health and let people deploy whenever it's suitable. It's not nice to have lots of branch for each environment, impossible to keep them in sync.

PS. Keep in mind that, deployments are not deploying only image tag changes, but also new K8S resources. So auto deployment tricks won't work.

What do you guys think?

[stefanprodan]

Flux v2 will support sync from a Git tag and it will also be able to follow semver releases, e.g.:

apiVersion: source.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
  name: podinfo
spec:
  interval: 1m
  url: https://github.com/stefanprodan/podinfo
  ref:
    tag: 4.0.0 # <- fix tag
---
apiVersion: source.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
  name: podinfo
spec:
  interval: 1m
  url: https://github.com/stefanprodan/podinfo
  ref:
    semver: ">=4.0.0 <5.0.0" # <- semver range

More info here: https://toolkit.fluxcd.io/

[kingdonb]

Since this feature is complicated and already supported in Flux v2, I am closing this as new features like this won't be backported to Flux v1 which is in maintenance mode. Glad to see another use case that is supported now, thanks to redesign!