HelmRelease doesnt delete prev resources on update

[pdeva]

Describe the bug

I created a HelmRelease test-flux with a deployment, service and image all named test-flux. I initially gave it correct values in values.yaml and it ran successfully. Then i gave it incorrect values , eg test-fluxa instead of test-flux for the image name, app name etc. Surely enough it resulted in the pods going in an error state. All looked good so far.

However, when i reverted the changes back (to test-flux), even though HelmRelease spun up the new successful pods. It didnt delete the previous service and deployments (named test-fluxa).

kubectl get all --all-namespaces -l='app.kubernetes.io/managed-by=Helm,helm.toolkit.fluxcd.io/name=test-flux'
NAMESPACE   NAME                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
default     service/app-test-flux    ClusterIP   10.100.104.242   <none>        80/TCP    3d19h
default     service/app-test-fluxa   ClusterIP   10.100.138.70    <none>        80/TCP    3d19h

NAMESPACE   NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
default     deployment.apps/app-test-flux    2/2     2            2           3d19h
default     deployment.apps/app-test-fluxa   0/2     2            0           3d19h

Steps to reproduce

1. Create HelmRelease with successfull service and deployment.
2. Change values so service name, deployment differ and deployment is in error state.
3. Fix service name, deployment name, etc back to original value.
4. Previous faulty services, deployments are not deleted

Expected behavior

Previously generated resources of a helm release should always be deleted when resource names are updated.

Screenshots and recordings

No response

OS / Distro

Ubuntu 22.04

Flux version

v0.31.2

Flux check

► checking prerequisites
✗ flux 0.31.3 <0.31.4 (new version is available, please upgrade)
✔ Kubernetes 1.21.12-eks-a64ea69 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.22.1
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.23.4
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.19.2
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.26.2
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.24.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.25.9
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta1
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1beta2
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta1
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta1
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta1
✔ receivers.notification.toolkit.fluxcd.io/v1beta1
✔ all checks passed

Git provider

github

Container Registry provider

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[bernardgut]

I have the same issue. Basically :

1. create a helmRelease with flux create helmrelease postgres --chart postgresql --source HelmRepository/bitnami ..., commit changes and watch reconciliation
2. modify the above manifest and add something like spec.releaseName: someUpdatedReleaseName, commit and watch reconciliation
3. You now have one postgres helmRelease but when you do helm list you see two helm release deployed: both the old release and the new one. If you check the namespace, all resources are deployed twice.

How can I automate garbage collection of dangling helm resources ? @pdeva did you ever manage to find a mitigation to this ?

thanks

[stefanprodan]

Fixed in fluxcd/helm-controller#738

[MartinEmrich]

I have a possibly related phenomenon here: A Helm chart introduced some ressources (a deployment, a service) in one of the previous versions, and removed them again later.
The image referenced in the deployment was never available.

But the resources are still present on the cluster. (Deployment, with the Pods being in ImagePullBackOff).

HelmRelease claims to be successfully reconciled. I set driftDetection.mode to "warn", with no effect.
flux trace still recognizes the surplus deployment as connected to the HelmRelease.
Manual flux reconcile helmrelease has no effect.

Pulling and helm template -ing the chart locally confirms that the Deployment is no longer part of the Chart output.

Versions:

flux: v2.3.0
distribution: flux-v2.3.0
helm-controller: v1.0.1
image-automation-controller: v0.38.0
image-reflector-controller: v0.32.0
kustomize-controller: v1.3.0
notification-controller: v1.3.0
source-controller: v1.3.0

Is the helm-controller supposed to remove resources from the cluster if they get removed in a newer chart version?