HelmRelease reconciliation failed if helm-controller pod recreated

[KosShutenko]

Describe the bug

We have seven k8s clusters with FluxCD ("--concurrent=16" args for helm and customise controllers).
We found issue when helm-controller pod is recreated - all HelmReleases which where in installation (upgrade) process going to "False" status with message "Helm upgrade failed: another operation (install/upgrade/rollback) is in progress"

Steps to reproduce

1. Setup k8s cluster and bootstrap flux.
2. Install some helm charts via HelmReleases.
3. Wait for message "Release reconciliation succeeded"
4. Start upgrading helm chart to newest version (try to make update with new image)
5. Kill helm-controller-**- pod in flux-system namespace.
6. Wait when new pod will be up and running.
7. Check helm releases

After that we cannot reconcile/upgrade/etc. this helm release.
All you can do its remove and install it from the scratch.

Expected behavior

Expected that new pod with helm-controller will continue reconciliation, and change status to "True" with message "Release reconciliation succeeded".

Screenshots and recordings

No response

OS / Distro

macOS 13.2.1

Flux version

v0.37.0

Flux check

► checking prerequisites
✗ flux 0.37.0 <0.41.1 (new version is available, please upgrade)
✔ Kubernetes 1.23.16-gke.1100 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.27.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.31.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.29.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.32.1
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta1
✔ buckets.source.toolkit.fluxcd.io/v1beta1
✔ gitrepositories.source.toolkit.fluxcd.io/v1beta1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta1
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta2
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta1
✔ receivers.notification.toolkit.fluxcd.io/v1beta1
✔ all checks passed

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[hiddeco]

Please update to Flux >=v0.41.0, which comes with the changes described in fluxcd/helm-controller#644.

[KosShutenko]

Hello,

@hiddeco thank you!
I've updated Flux to 0.41.2 and bootstrapped all clusters. After that I've tried to reproduce issue with "Helm upgrade failed: another operation (install/upgrade/rollback) is in progress", and now its gone.

Can confirm that upgrade is solving this issue.

Thank you.