unexpected 'pending-install' status of helm release when using different storage namespace and target namespace

[zxkane]

I meet an issue of Helm release, the flux indeed successfully deployed the helm release(pod is healthy/svc is working) in cluster, however the helm always returns the status of release as pending-install. for example,

$ helm ls -A --filter pod --pending
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
podinfo app 1 2022-03-18 11:55:29.438559647 +0000 UTC pending-install podinfo-6.1.0 6.1.0

$ flux get hr -n app
NAME READY MESSAGE REVISION SUSPENDED
podinfo True Release reconciliation succeeded 6.1.0 False

the side effect is the subsequent update of helm release will fail due to the helm considers the release in existing install/upgrade operations.

I suspect it’s caused by using different storage namespace for helm release, looks like it works for flux to deploy the release, however the helm can not retrieve the release history information if the storage is not in the namespace of release.

You can reproduce this behavior with below manifest in your repo.

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: helm-controller-issue
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/zxkane/flux-helm-storage-namespace-issue.git
  ref:
    branch: main  
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: helm-controller-issue-infra
  namespace: flux-system
spec:
  interval: 10m0s
  prune: true
  sourceRef:
    kind: GitRepository
    name: helm-controller-issue
    namespace: flux-system
  path: ./infra
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: helm-controller-issue-app
  namespace: app
spec:
  serviceAccountName: app-reconciler
  interval: 10m0s
  prune: true
  sourceRef:
    kind: GitRepository
    name: helm-controller-issue
    namespace: flux-system
  path: ./app
  dependsOn:
    - name: helm-controller-issue-infra
      namespace: flux-system
  patches:
    - patch: |-
        - op: replace
          path: /spec/serviceAccountName
          value: app-reconciler
        - op: replace
          path: /spec/storageNamespace
          value: helm-storage
      target:
        group: helm.toolkit.fluxcd.io
        version: v2beta1
        kind: HelmRelease
    - patch: |-
        - op: replace
          path: /spec/serviceAccountName
          value: app-reconciler
      target:
        group: kustomize.toolkit.fluxcd.io
        version: v1beta2
        kind: Kustomization

[hiddeco]

With this combination, using helm is indeed not possible, but the controller operations will continue to work.

[zxkane]

Should it be something address in upstream helm operator? It means the field `storageName` of Helm release useless and error prone. Hidde Beydals ***@***.***> 于 2022年3月18日周五 下午9:19写道：

…

With this combination, using helm is indeed not possible, but the controller operations will continue to work. — Reply to this email directly, view it on GitHub <#439 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGN4JYT2VX6PO3FDBUAMG3VAR7HZANCNFSM5RBWPP6Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[hiddeco]

It is not useless for the controller itself, as operations will succeed and it works well for some edge-case multi-tenancy scenarios. We could maybe do a better job documenting the behavior however.

[zxkane]

It's my fault. The pending-install status was caused by the update op of secrets was missing in helm storage namespace.

But looks like the failure of updating helm storage secrets was not captured by Helm controller and updating the HelmRelease accordingly. I did not find any info from the logs of helm-controller.