[bitnami/keycloak] Upgrade from 6.2.5 to 7.0.0 leads to postgres crashloopBackoff

[AlexisDucastel]

Name and Version

bitnami/keycloak 7.0.0

What steps will reproduce the bug?

We have a keycloak that was running with chart 6.2.5, we tried to upgrade to 7.0.0

By digging information on 7.0.0 commit diff and bitnami/postgres documentation, we achieved to proceed to upgrade on helm side. For this we deleted postgresql statefulset and postgresql secret according to Postgresql chart upgrade path from v10 to v11 (cf https://docs.bitnami.com/kubernetes/infrastructure/postgresql/administration/upgrade/#upgrading-instructions), and set values in keycloak release like following :

...
postgresql:
  auth:
    password: <value_from_secret["keycloak-postgresql"].data.postgresql-password>
    postgresPassword: <value_from_secret["keycloak-postgresql"]:postgresql-postgres-password>
  # we have to also let old value as the chart will complain if we do remove it before upgrade
  postgresqlPassword: <value_from_secret["keycloak-postgresql"]:postgresql-password>
...

Right now Helm is not throwing errors anymore, but Postgresql statefulset is in crashloopbackoff.
Here is the log of the postgres container :

postgresql 23:29:44.97 INFO ==> ** Starting PostgreSQL **
2022-03-03 23:29:44.982 GMT [1] FATAL: database files are incompatible with server
2022-03-03 23:29:44.982 GMT [1] DETAIL: The data directory was initialized by PostgreSQL version 11, which is not compatible with this version 14.2.

Are you using any custom parameters or values?

...
postgresql:
  auth:
    password: <value_from_secret["keycloak-postgresql"].data.postgresql-password>
    postgresPassword: <value_from_secret["keycloak-postgresql"]:postgresql-postgres-password>
  # we have to also let old value as the chart will complain if we do remove it before upgrade
  postgresqlPassword: <value_from_secret["keycloak-postgresql"]:postgresql-password>
...

What is the expected behavior?

We would expect postgresql to upgrade from previous chart version to the new one successfully

What do you see instead?

Postgresql subchart is deploying a postgresql version that seems to be not compatible with previous postgresql data files

Additional information

Just for a bit of context : We are using keycloak on a Rancher RKE2 kubernetes cluster v1.22. Persistence is enabled and Postgres PVC storage backend is a Rook Ceph RBD.

[AlexisDucastel]

For your information, we achieved to get Keycloak back by forcing posgresql image tag to 11.15.0-debian-10-r19 by deleting postgresql sts with

kubectl delete sts ibd-keycloak-postgresql -n keycloak --cascade=false

And then adding this to values :

postgresql:
  image:
    tag: 11.15.0-debian-10-r19

But this is clearly a workaround, as we are in chart bitnami/keycloak:7.0.0, but forcing subchart bitnami/postgresql with a tag that is not the default one of the subchart.

Maybe it should be useful to give an upgrade path to users to get from 6.X.X to 7.0.0 on bitnami/keycloak README ?

[juan131]

Hi @AlexisDucastel

We should've definitely include better upgrade instructions... Could you please follow these instructions and let us know if they work (the following example assumes that the release name is keycloak and the release namespace default)?

NOTE: Please, create a backup of your database before running any of those actions.

1. Obtain the credentials and the names of the PVCs used to hold the data on your current release:

export KEYCLOAK_PASSWORD=$(kubectl get secret --namespace default keycloak -o jsonpath="{.data.admin-password}" | base64 --decode)
export POSTGRESQL_PASSWORD=$(kubectl get secret --namespace default keycloak-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
export POSTGRESQL_PVC=$(kubectl get pvc -l app.kubernetes.io/instance=keycloak,app.kubernetes.io/name=postgresql,role=primary -o jsonpath="{.items[0].metadata.name}")

2. Delete the PostgreSQL statefulset (notice the option --cascade=false) and secret:

kubectl delete statefulsets.apps --cascade=false keycloak-postgresql
kubectl delete secret keycloak-postgresql --namespace default

3. Upgrade your release using the same PostgreSQL version:

CURRENT_PG_VERSION=$(kubectl exec keycloak-postgresql-0 -- bash -c 'echo $BITNAMI_IMAGE_VERSION')
helm upgrade keycloak bitnami/keycloak \
  --set adminPassword=$KEYCLOAK_PASSWORD \
  --set postgresql.image.tag=$CURRENT_PG_VERSION \
  --set postgresql.auth.password=$POSTGRESQL_PASSWORD \
  --set postgresql.persistence.existingClaim=$POSTGRESQL_PVC

4. Delete the existing PostgreSQL pods and the new statefulset will create a new one:

kubectl delete pod keycloak-postgresql-0

[foal]

I tried the proposed steps and they work, with notes bellow:

1. I have another namespace so I will need to change/add missing parameters
2. helm update ask me to use global.postgresql.auth.postgresPassword parameter

kot@debian:~# CURRENT_PG_VERSION=$(kubectl exec --namespace keycloak keycloak-postgresql-0 -- bash -c 'echo $BITNAMI_IMAGE_VERSION')
helm upgrade keycloak bitnami/keycloak \
  --namespace keycloak \
  --set adminPassword=$KEYCLOAK_PASSWORD \
  --set postgresql.image.tag=$CURRENT_PG_VERSION \
  --set postgresql.auth.password=$POSTGRESQL_PASSWORD \
  --set postgresql.persistence.existingClaim=$POSTGRESQL_PVC
Error: UPGRADE FAILED: execution error at (keycloak/charts/postgresql/templates/secrets.yaml:17:24):
PASSWORDS ERROR: You must provide your current passwords when upgrading the release.
                 Note that even after reinstallation, old credentials may be needed as they may be kept in persistent volume claims.
                 Further information can be obtained at https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases

    'global.postgresql.auth.postgresPassword' must not be empty, please add '--set global.postgresql.auth.postgresPassword=$POSTGRES_PASSWORD' to the command. To get the current value:

        export POSTGRES_PASSWORD=$(kubectl get secret --namespace "keycloak" keycloak-postgresql -o jsonpath="{.data.postgres-password}" | base64 --decode)

kot@debian:~# CURRENT_PG_VERSION=$(kubectl exec --namespace keycloak keycloak-postgresql-0 -- bash -c 'echo $BITNAMI_IMAGE_VERSION')
helm upgrade keycloak bitnami/keycloak \
  --namespace keycloak \
  --set adminPassword=$KEYCLOAK_PASSWORD \
  --set postgresql.image.tag=$CURRENT_PG_VERSION \
  --set postgresql.auth.password=$POSTGRESQL_PASSWORD \
  --set postgresql.persistence.existingClaim=$POSTGRESQL_PVC \
  --set global.postgresql.auth.postgresPassword=$POSTGRESQL_PASSWORD
Release "keycloak" has been upgraded. Happy Helming!

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.