Document solution to problem of random passwords change on upgrade (from helm/charts)

[scottrigby]

I was replying in helm/charts#3591 and realized this is a much bigger problem that deserves a charts issue to decide whether we should agree on and document a policy for this across charts.

Can we address this at one of the next charts meetings?

Related:

• [stable/mysql] Update mysql user passwords on secrets change charts#3591
• [stable/percona] Support the new {{ .Secrets.password }} management charts#4258
• feat(2196): secrets management  helm#2721
• Secret Management in Helm helm#2196
• [incubator/keycloak] DB and Admin user secrets overwritten when upgrading charts#5129
• [stable/redis] helm upgrade change password secret charts#4472
• Add doc about redis data only persistence charts#4965
• Redis updates secret to invalid password charts#2538
• [patroni] Support updating database credentials via chart upgrade charts#2636
• Problems with randAlphaNum and upgrades helm#3053
• https://github.com/kubernetes/helm/blob/master/docs/charts_tips_and_tricks.md#automatically-roll-deployments-when-configmaps-or-secrets-change
• mongodb chart not able to run charts#2488

[unguiculus]

When I became a maintainer I learned it was best practice to auto-generate passwords and have told folks to follow this pattern. I never really questioned this but am not really happy about it either. I appreciate you brought up the discussion.

Auto-generating passwords is kind of nice. It makes it easy to get started and is certainly better than a hard-coded password. But for a real-world installation, I wouldn't rely on this. So, we need at least better documentation.

Thinking about it again, wouldn't it be better not to provide a default password at all? We could use the required function forcing the user to explicitly set a password. For CI, a custom values file could provide a password.

[unguiculus]

cc @kubernetes/charts-maintainers

[alexvicegrab]

This is indeed a problem. Currently I'm overcoming this by specifying a value with --set flag to override sensitive information (e.g. passwords) during the upgrade process.

I tend to save charts on git repositories but without the password. Ideally I neither want to:

1. Cause bloat, by saving one copy of each chart values.yaml for each (e.g.) PostGres chart I install, nor:
2. Cause security issues by saving passwords in Git repositories
3. Cause high complexity (what I currently do) by having to retrieve the passwords from the relevant Helm deployment secrets and resupplying them in the update command.

A way to automate 3 in a K8S/Helm way would go a long way.

[alexvicegrab]

e.g. it could be useful to add an annotation that prevents updating of a secret like this:

"helm.sh/resource-policy": keep

Except this only works for preventing deletion.

[geekuillaume]

I had the same problem. You can create a secret only on install by using a pre-install hook. The secret will be left untouched on upgrades. The only problem is that it's not managed by Helm and so not deleted when you delete the release.

[michaelfig]

I think the issue is not feeding back secrets into a chart upgrade, nor a problem with generating passwords. I think the issue is that if a chart is generating a random password, it should not overwrite an existing secret.

This seems like it should use the following wrapper around the secret definition:

{{- if or .Values.somePassword .Release.IsInstall}}
apiVersion: v1
kind: Secret
...
data:
  {{- if .Values.somePassword }}
  some-password: {{ .Values.somePassword | b64enc | quote }}
  {{- else }}
  some-password: {{ randAlphaNum 10 | b64enc | quote }}
  {{- end }}
{{- end}}

Thoughts?

[alexvicegrab]

@michaelfig I like this, either we explicitly specify the password, or we autogenerate it during an install.

Does this work? (i.e. Have you tried running this to check that it does not entirely delete the secret during an update when not finding the secret template?)

[alexvicegrab]

Tried it myself just now, alas Helm will simply wipe out the secret.

[achempion]

The service is writing password to the secrets. Maybe it will be better to check first for existence and use old if a secret has a value already.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[alexvicegrab]

Is there any progress on setting up a way for random secrets to not be reset implicitly during upgrades?

[bacongobbler]

From the dev call, one possible solution to try would be through an install hook that writes out a configmap.

[bacongobbler]

That or checking for the presence of .Release.IsUpgrade.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[yurrriq]

Any update?