Requested feature: flux OCI helm repositories notice when tags on remote registry change

[gfichtenholt]

Filing an issue per souleb's request in https://cloud-native.slack.com/archives/CLAJ40HV3/p1658304872546299
Hello Flux gurus,
is the following a bug or a feature?

I have a public OCI helm registry on ghcr.io with URL oci://ghcr.io/gfichtenholt/helm-charts
That registry has one package podinfo and it has two versions 6.1.6 and 6.1.5
Sanity check:

$ helm pull oci://ghcr.io/gfichtenholt/helm-charts/podinfo --version 6.1.6
Pulled: ghcr.io/gfichtenholt/helm-charts/podinfo:6.1.6
Digest: sha256:7d05ef40f1007f895ed04ed11da11dfcc369217b7c5365e7ee80d0f636bd6d88
$ helm pull oci://ghcr.io/gfichtenholt/helm-charts/podinfo --version 6.1.5
Pulled: ghcr.io/gfichtenholt/helm-charts/podinfo:6.1.5
Digest: sha256:80e6d2e7f6d21800621530fc4c5b70d0e458f11b2c05386ea5d058c4e86d6e93
$ helm pull oci://ghcr.io/gfichtenholt/helm-charts/podinfo --version 6.1.4
Error: ghcr.io/gfichtenholt/helm-charts/podinfo:6.1.4: not found

I then register a flux OCI helm repo object

$ kubectl get helmrepository -o yaml
apiVersion: v1
items:
- apiVersion: source.toolkit.fluxcd.io/v1beta2
  kind: HelmRepository
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"source.toolkit.fluxcd.io/v1beta2","kind":"HelmRepository","metadata":{"annotations":{},"name":"podinfo","namespace":"default"},"spec":{"interval":"5s","secretRef":{"name":"oci-creds"},"type":"oci","url":"oci://ghcr.io/gfichtenholt/helm-charts"}}
    creationTimestamp: "2022-07-20T08:02:53Z"
    finalizers:
    - finalizers.fluxcd.io
    generation: 1
    name: podinfo
    namespace: default
    resourceVersion: "171547"
    uid: 87afda21-a56b-40a6-ac72-f793395105d4
  spec:
    interval: 5s
    secretRef:
      name: oci-creds
    timeout: 60s
    type: oci
    url: oci://ghcr.io/gfichtenholt/helm-charts
  status:
    conditions:
    - lastTransitionTime: "2022-07-20T08:02:54Z"
      message: Helm repository is ready
      observedGeneration: 1
      reason: Succeeded
      status: "True"
      type: Ready
    observedGeneration: 1
kind: List
metadata:
  resourceVersion: ""

All is good. I then push a version of the chart that doesn't already exist on the server

$ helm registry login ghcr.io -u $GITHUB_USER -p $GITHUB_TOKEN 
WARNING: Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded
$ helm push testdata/charts/podinfo-6.1.4.tgz oci://ghcr.io/gfichtenholt/helm-charts
Pushed: ghcr.io/gfichtenholt/helm-charts/podinfo:6.1.4
Digest: sha256:ad1ced1a4bf77e880333376ccec037e2fc6271a361ed431c07dadd9552b9af10
$ helm registry logout ghcr.io
Removing login credentials for ghcr.io

I expect at some point (after specified interval) flux reconciler will kick in and fire a MODIFIED event. Lets say I have a watcher running, e.g. via
$ kubectl get helmrepository --watch
I wait a few minutes but it does not happen. Is this a bug or a feature?
Thank you in advance
Greg

[gfichtenholt]

additional notes:

• to answer the question "would having last x tags incrementally in the repository status works?"
  I don't understand why I need to specify any number of tags. Lets just say I'd like to be notified when any tag is added or deleted. I mean you keep track of whatever you need to keep track of (its your CRD and your status to maintain). But as a user I don't want to worry about it

[hiddeco]

As per RFC, the HelmRepository of type OCI does not keep track of changes to the repository. This is inline with the changes Helm made around the introduction of OCI, which are documented here and state:

The chart cache has been removed (no helm chart list etc.)

Instead, it provides an authentication reference which is then utilized by a HelmChart object to see if there is a new chart for a provided SemVer constraint. This object will produce an event (and produce a new artifact).

Keeping track of individual tags is out of scope for the source-controller at present. If the user wants automation of a HelmChart as explicit version declaration, they can make use of the image-automation-controller which already contains the more complex logic of keeping track of image tags. This should work fine for OCI images while using the markers for the OCI chart values instead. Improved documentation of this is being worked on.

I think this is not likely to change before the distribution spec offers more advanced query capabilities, as the current way you have to collect information from tags and manifests (to inspect e.g. on media type) is expensive in terms of number of requests you have to make to maintain this index. Which is why almost all solutions currently assume any tag to be of the requested type, and then assert it during pull (which is essentially what the HelmChart object does as well).

What are your more precise requirements? I may help you find an alternative solution, or steer you into the direction of a feature request which seems doable short term.

[gfichtenholt]

Thank you for your comment. The "precise" requirement (again remember I am not an end user) is that Helm OCI charts notice the changes on remote and record it locally. Same way that regular HTTP charts notice changes to remote index.yaml and record this fact in the HelmRepository CRD via status.artifact.checksum field. ref https://fluxcd.io/docs/components/source/helmrepositories/#status Without this feature, flux OCI support is missing an important feature and lacks parity with support for HTTP repos, IMO. Being able to install HelmReleases from OCI charts is a good start but. There is no catalog (no demand or cached), and you're saying flux won't keep track of contents of remote at all outside of user interaction. I understand this may be out of scope in the RFC and out of scope for the short term. I did read the link to helm deprecated features. Seems I am asking for a feature that helm chose to get rid of. Disappointing for sure. Anyway, I understand my feedback is a few months late. If I can't change your mind then so be it, of course I will accept it and move on.
I will also look into image-automation-controller as I have not used it before.

P.S. I read https://github.com/helm/community/blob/main/hips/hip-0006.md#5-cache-is-removed and that argument is very subjective and not backed up by any evidence. I disagree with just about every sentence. Bit that's on helm

P.P.S. I also recognize the fact that in #805 I asked flux to be consistent with helm CLI and here I am arguing for flux to go beyond helm CLI (in other words, be inconsistent with helm CLI). The goal for flux to be at least as useful as helm CLI or even more useful in this case. IMO, of course

[gfichtenholt]

FWIW, I confirm the following use case is working as expected:

1. remote OCI repo contains only podinfo version 6.1.5
2. create flux HelmRepository for (1)
3. create flux HelmRelease for (2) results in podinfo version 6.1.5 deployed
4. push podinfo version 6.1.6 to (1)
5. wait for specified repo interval + a bit
6. now (3) is automatically upgraded to 6.1.6

Arguably, this is the most important use case I have come up with, and it works fine. So, just to hammer the point again, whichever direction you take on this issue I will accept. Thanks

[makkes]

@gfichtenholt looking back at the discussion there's 2 potential paths forward with this:

1. Employ Flux's image automation to be notified of new chart versions and build upon that.
2. Submit an RFC proposing a solution to be implemented in Flux, probably preceded by a GitHub discussion.

As per your most recent comment, I'm going to close this issue now.

[gfichtenholt]

@gfichtenholt looking back at the discussion there's 2 potential paths forward with this:

1. Employ [Flux's image automation](https://fluxcd.io/docs/components/image/) to be notified of new chart versions and build upon that.

2. Submit [an RFC](https://github.com/fluxcd/flux2/tree/main/rfcs) proposing a solution to be implemented in Flux, probably preceded by a GitHub discussion.

As per your most recent comment, I'm going to close this issue now.

Sounds good. Thank you