Upgrade to version 2.1.0 broken using only nautobot.secretKey

[ffernandes22]

Recent changes addressing the use of existing secrets #364 #369 included in version 2.1.0 broke the usage of nautobot.secretKey:

• Test using helm diff:

$ helm diff upgrade nautobot nautobot/nautobot --version=2.1.0 --reuse-values
Error: Failed to render chart: exit status 1: Error: execution error at (nautobot/templates/secret.yaml:17:8): 
PASSWORDS ERROR: You must provide your current passwords when upgrading the release.
                 Note that even after reinstallation, old credentials may be needed as they may be kept in persistent volume claims.
                 Further information can be obtained at https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases

    '' must not be empty, please add '--set =$NAUTOBOT_SECRET_KEY' to the command. To get the current value:

        export NAUTOBOT_SECRET_KEY=$(kubectl get secret --namespace "nautobot" nautobot-env -o jsonpath="{.data.NAUTOBOT_SECRET_KEY}" | base64 -d)


Use --debug flag to render out invalid YAML

Error: plugin "diff" exited with error

• Doing helm upgrade:

helm upgrade -n nautobot nautobot nautobot/nautobot --reuse-values --version=2.1.0
Error: UPGRADE FAILED: template: nautobot/templates/secret.yaml:17:8: executing "nautobot/templates/secret.yaml" at <include "nautobot.secret.env" .>: error calling include: template: nautobot/templates/_secrets.tpl:2:18: executing "nautobot.secret.env" at <.Values.nautobot.django.existingSecret>: nil pointer evaluating interface {}.existingSecret

Setting variable nautobot.django.secretKey does not seem to be the solution (nautobot.django.secretKey: "my-secret-key"):

helm diff upgrade nautobot nautobot/nautobot --version=2.1.0 --reset-values -f=nautobot/values.yaml 
Error: Failed to render chart: exit status 1: Error: execution error at (nautobot/templates/secret.yaml:17:8): 
PASSWORDS ERROR: You must provide your current passwords when upgrading the release.
                 Note that even after reinstallation, old credentials may be needed as they may be kept in persistent volume claims.
                 Further information can be obtained at https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases

    'my-secret-key' must not be empty, please add '--set my-secret-key=$NAUTOBOT_SECRET_KEY' to the command. To get the current value:

        export NAUTOBOT_SECRET_KEY=$(kubectl get secret --namespace "nautobot" nautobot-env -o jsonpath="{.data.NAUTOBOT_SECRET_KEY}" | base64 -d)


Use --debug flag to render out invalid YAML

Error: plugin "diff" exited with error

Also, reference documentation for latest/stable branch seems to be outdated (missing at least nautobot.django.* settings).

[m4rg4sh]

@ffernandes22 I've authored most of that change but we use argocd ourselves and I did not test the upgrade procedure extensively since I don't have a lot of experience with it (e.g. using --reuse-values). If you can attach the command and values you used to install the chart I could check it out later this week :)
@ubajze I've added the mentioned references and ran the pre-commit commands to update the docs. Do you know why this is not visible after the merge to stable?

[m4rg4sh]

Hi @ffernandes22
I've quickly looked into this.
If you specify --reuse-values you try to use the values.yaml from the previous version with the current chart's logic. That doesn't work because the previous version doesn't include the nautobot.django.* keys.

These are by default set to empty string (and thus ignored) but still checked. By specifying --reuse-values you tell helm to not use those new defaults and that's why you get a null pointer exception.

I think if you use --reset-then-reuse-values instead you should get the desired result, at least it worked with my limited testing.

[ffernandes22]

Hi @m4rg4sh, thanks for the quick reply!

Actually I used kubernetes.core.helm ansible module to deploy and update our setup. It always worked as expected but not this time and I failed to mimic the commands that the module performs:

It starts with the first helm diff that I pasted and immediately failed. And when I tried to directly issue an helm upgrade I kept the flag --reuse-values because I thought that the new charm default values were always introduced. As you said, this is not true and that's the reason why my helm upgrade was failing.

Using either --reset-then-reuse-values or --reset-values -f=nautobot/values.yaml --set-file=nautobot.config=nautobot/nautobot_config.py works as expected.

However, doing a similar helm diff instead of helm upgrade still do not work, both before and after (successfully) upgrading the release to version 2.1.0.

$ helm diff upgrade nautobot nautobot/nautobot --version=2.1.0 --reset-values -f=nautobot/values.yaml --set-file=nautobot.config=nautobot/nautobot_config.py
Error: Failed to render chart: exit status 1: Error: execution error at (nautobot/templates/secret.yaml:17:8): 
PASSWORDS ERROR: You must provide your current passwords when upgrading the release.
                 Note that even after reinstallation, old credentials may be needed as they may be kept in persistent volume claims.
                 Further information can be obtained at https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases

    '' must not be empty, please add '--set =$NAUTOBOT_SECRET_KEY' to the command. To get the current value:

        export NAUTOBOT_SECRET_KEY=$(kubectl get secret --namespace "nautobot" nautobot-env -o jsonpath="{.data.NAUTOBOT_SECRET_KEY}" | base64 -d)


Use --debug flag to render out invalid YAML

Error: plugin "diff" exited with error

This is probably an issue with helm diff plugin, I will check for similar issues.

[m4rg4sh]

Hi @ffernandes22

I think it's also worth noting that the error when using helm diff refers to the bitnami docs.
The nautobot charts use bitnami subcharts for postgresql and redis, etc as well as some password logic from the bitnami "common" charts.
I think it is reasonable to assume that this is not exactly a nautobot-specific issue but that you'd have similar issues with this plugin and bitnami charts in general.

[ffernandes22]

I can agree with you @m4rg4sh. However it seems to be related to how nautobot is using bitnami "common" chart.

I've not yet understood why (I have little experience with Helm Charts) but there is occurring some validation against the value of nautobot.django.secretKey. If I do not define it, inheriting the default '' value, the error is:

'' must not be empty, please add '--set =$NAUTOBOT_SECRET_KEY' to the command. To get the current value:`

If i set nautobot.django.secretKey: "some-random-value":

'some-random-value' must not be empty, please add '--set some-random-value=$NAUTOBOT_SECRET_KEY' to the command. To get the current value:`

I will try to find out what is causing this, if you have any hint, I appreciate.

However my main motivation to open this issue is clarified and solved so feel free to close the issue and, once again, thanks for the quick assist 👌
(It would still be good to solve the not updated docs thing)

[m4rg4sh]

Aah I see your point now @ffernandes22.

I've dug a little deeper and I think I found the bug.

I'm running out of time for today but I'll get back to you on this.

[m4rg4sh]

@ffernandes22 I've actually made a mistake in the code, it used the value specified at the given dictionary key instead of the name of the key itself 🤦

[m4rg4sh]

@ubajze only thing left for you is to review the PR and check why the stable docs on the website aren't up to date.
The reference.md in the repo looks good.

[ffernandes22]

Hi @m4rg4sh
Great! Thanks for the debug 🥇