PostgreSQL and Redis existingSecret not respected on ArgoCD sync

[salcinad]

The Helm chart version

5.0.0-beta.167

Environment Versions

Kubernetes: 1.28.8
ArgoCD: v2.13.0+347f221

Custom chart values

        superuser:
          name: admin
          email: example@example.com
          existingSecret: "netbox-superuser"
        existingSecret: "netbox-config"
        postgresql:
          auth:
            existingSecret: "netbox-postgresql"
            secretKeys:
              adminPasswordKey: "postgres-password"
              userPasswordKey: "password"
        redis:
          auth:
            existingSecret: "netbox-redis"
            existingSecretPasswordKey: "redis-password"

Current Behavior & Steps to Reproduce

Tested with different option in valuey.yaml for existing secrets but did it just do not work. Once the secret is overwritten, the worker complains that password is incorrect, and the whole netbox pod does not start.
django.db.utils.OperationalError: connection failed: connection to server at "10.95.18.196", port 5432 failed: FATAL: password authentication failed for user "netbox"

Even if we do not use existingSecrets and let ArgoCD create secrets, they are getting regenerated over every ArgoCD sync. No issue with other applications.

Expected Behavior

The existingSecrets should not be overwriten by netbox chart.

NetBox Logs

Not an NetBox issue.

[LeoColomb]

Thanks for filing this issue, @salcinad
I see this is a direct follow-up of #420.

Did you manage to get one deployment working? If not, I'd advocate to uninstall completely the chart and try again in a proper deployment.

Even if we do not use existingSecrets and let ArgoCD create secrets, they are getting regenerated over every ArgoCD sync.

That is very surprising, as password management is handled by the chart itself and should keep any previously defined/generated value.
And this is probably an issue on ArgoCD or on your configuration.

The existingSecrets should not be overwriten by netbox chart.

Can you share the secrets as defined in your Kubernetes cluster, before and after an upgrade or a sync? You can, of course, change the actual values before posting in here, but please keep a way to see if the values are the same or different between versions of the secrets.

[salcinad]

I see this is a direct follow-up of #420.

Indeed, as that Issue is closed, I did not realize that, sorry for bumping closed issue.

Did you manage to get one deployment working? If not, I'd advocate to uninstall completely the chart and try again in a proper deployment.

First deployment is always Ok. But any other update, it would like to sync netbox-postgres and netbox-redis secrets, other two secrets (netbox-config, netbox-superuser) are fine. And when testing I am aways deleting Application and the namespace on k8s cluster so everything is clear.

That is very surprising, as password management is handled by the chart itself and should keep any previously defined/generated value.
And this is probably an issue on ArgoCD or on your configuration.

This can ofcourse be an ArgoCD issue on our side, but we also have same bitnami chart for standalone postgres and not having this issue, or for example Harbor is also using redis as sub chart (same bitnami chart) and no issue there. We are still looking, but noting obivisly is popuig up.

Can you share the secrets as defined in your Kubernetes cluster, before and after an upgrade or a sync? You can, of course, change the actual values before posting in here, but please keep a way to see if the values are the same or different between versions of the secrets.

Sure, will test tomorrow in different cluster and let provided the info here.

Edit # 1
When I delete the Application in ArgoCD and then also the namespace where netbox is deployed I see following in ArgoCD, even we have Secrets (sealed-secrets, but also tested without sealed-secrets). But even I don't have secrets, and have values.yaml as in first post then pod should complain about missing secrets and not creating its own.

/Secret/netbox/netbox-postgresql
apiVersion: v1
data:
  password: ++++++++
  postgres-password: ++++++++
kind: Secret
metadata:
  labels:
    app.kubernetes.io/instance: netbox
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: postgresql
    app.kubernetes.io/version: 17.2.0
    argocd.argoproj.io/instance: netbox-dev
    helm.sh/chart: postgresql-16.3.1
  name: netbox-postgresql
  namespace: netbox
type: Opaque


/Secret/netbox/netbox-redis
apiVersion: v1
data:
  redis-password: ++++++++
kind: Secret
metadata:
  labels:
    app.kubernetes.io/instance: netbox
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: redis
    app.kubernetes.io/version: 7.4.1
    argocd.argoproj.io/instance: netbox-dev
    helm.sh/chart: redis-20.5.0
  name: netbox-redis
  namespace: netbox
type: Opaque

[LeoColomb]

Thanks for your input.
I'm going a little ahead and suggest some potential root causes.

• If you first generate secret with the chart and then reference them with existingSecret, this will not work: generated secrets will not be rendered anymore and thus not kept in the cluster, previous version will be deleted.
• If using auto-management inherited from the chart is not working as expected (i.e. passwords are not kept between upgrade/sync, which is surprising already but might happen), then you must declare first the secrets manually before referencing them into the values.

[salcinad]

• If you first generate secret with the chart and then reference them with existingSecret, this will not work: generated secrets will not be rendered anymore and thus not kept in the cluster, previous version will be deleted.

This is how we did Initial setup, we let chart deploy without setting existingSecret or setting passwords manually. then exported these secrets to yaml, deleted unneeded data from yaml (see below), deleted application, deleted namespace where it was installed, used then existingSecret to reference to these secrets exported yaml, and added them to values.yaml under extraDeploy , updated ArgoCD Application.

$ cat netbox-postgresql-dev.yaml
apiVersion: v1
data:
  password: bXB5bXSd05TVlid05TRQ==
  postgres-password: WG40FpbSlW9YWG40TQ==
kind: Secret
metadata:
  name: netbox-postgresql
  namespace: netbox
type: Opaque

$ cat netbox-redis-dev.yaml
apiVersion: v1
data:
  redis-password: TKZGbjVMW2mpQQ==
kind: Secret
metadata:
  name: netbox-redis
  namespace: netbox
type: Opaque

$

In my comment above the application deployed successfully, the ArgoCD deployed sealed-secrets which are applied to correct namespace in correct k8s cluster and NetBox is available over Ingress and login (superuser - which is also existing secret) its working just fine, all pods up (see below), as well as LDAPs.

$ k get pods -n netbox
NAME                                 READY   STATUS      RESTARTS      AGE
netbox-869547468c-6mdqv              1/1     Running     5 (9h ago)    10h
netbox-housekeeping-28906560-qnc9v   0/1     Completed   0             7h6m
netbox-postgresql-0                  1/1     Running     0             10h
netbox-redis-master-0                1/1     Running     0             10h
netbox-redis-replicas-0              1/1     Running     0             10h
netbox-redis-replicas-1              1/1     Running     0             10h
netbox-redis-replicas-2              1/1     Running     0             10h
netbox-worker-7cc9d556d7-5pffx       1/1     Running     2 (10h ago)   10h
$

But now ArgoCD is showing that this Application is OutOfSync, as there are these two secret which we did not sync in fist name. We also checked the secret in namespace with password in yaml files and are identical.
This is somehow weird, but we are out of Ideas what could be wrong.

[LeoColomb]

Thanks for the details.
Can you try following the same procedure but changing the name of the manual secrets?

1. Raw install
2. Secrets exports
3. Clear labels and change the name (e.g. with -2 prefix)
4. Add existingSecret with the new names

If this still does not work, can you confirm the secrets values do not get base64 encoded twice?