[BUG] - Keycloak does not start on the first try

[krassowski]

Describe the bug

Locally I always have to restart keycloak after reboot. I see the following:

Connecting to github.com (20.26.156.215:443)
wget: note: TLS certificate validation not implemented      
Connecting to objects.githubusercontent.com (185.199.109.133:443)   
wget: can't open '/data/keycloak-metrics-spi-2.5.3.jar': File exists       
Stream closed EOF for dev/keycloak-0 (initialize-spi-metrics-jar) 

nebari/src/_nebari/stages/kubernetes_keycloak/template/modules/kubernetes/keycloak-helm/values.yaml

Lines 26 to 40 in 2000996

	- command:
	- sh
	- -c
	- |
	wget https://github.com/aerogear/keycloak-metrics-spi/releases/download/2.5.3/keycloak-metrics-spi-2.5.3.jar -P /data/ &&
	export SHA256SUM=9b3f52f842a66dadf5ff3cc3a729b8e49042d32f84510a5d73d41a2e39f29a96 &&
	if ! (echo "$SHA256SUM /data/keycloak-metrics-spi-2.5.3.jar" | sha256sum -c)
	then
	echo "Error: Checksum not verified" && exit 1
	else
	chown 1000:1000 /data/keycloak-metrics-spi-2.5.3.jar &&
	chmod 777 /data/keycloak-metrics-spi-2.5.3.jar
	fi
	image: busybox:1.36
	name: initialize-spi-metrics-jar

This code was most recently modified in #1810 (CC @Adam-D-Lewis)

It also means that the jupyterhub pod does not start up: #2547

Expected behavior

Keycloak starts on the first try

OS and architecture in which you are running Nebari

Ubuntu

How to Reproduce the problem?

1. Deploy locally on Ubuntu
2. Reboot
3. See that keycloak is not up

Command output

No response

Versions and dependencies used.

No response

Compute environment

None

Integrations

No response

Anything else?

No response

[Adam-D-Lewis]

Locally I always have to restart keycloak after reboot.

I'm trying to understand the issue. I guess when you reboot, the k8s pods restart and run the init containers again, but somehow the file system of the keycloak pod is from the prior run? The PR I opened might fix it, but I haven't tested it.

[viniciusdc]

I am also not entirely sure; in theory, unless the keycloak metadata for the Kubernetes resource has changed, it should not restart. @krassowski why do you need it to restart? Also, because the helm provider handles this, some changes made to the values.yaml, do not trigger a change in the pod. This is a bug with the provider itself, as it can be seen in some other cases (I think this is related to how the provider calculates the diff)

[krassowski]

I need to restart it because when I boot my laptop it is always dead, I do not recall if it stuck in a restart loop or not but simply put nebari does not work until I get to k9s and restart it manually. I applied @Adam-D-Lewis's patch and will let you know if this helped or not as soon as I restart my laptop (but I think that patch is useful anyways because even if it is not the root cause, the logs will now be cleaner).

[krassowski]

Also, because the helm provider handles this, some changes made to the values.yaml, do not trigger a change in the pod. This is a bug with the provider itself, as it can be seen in some other cases (I think this is related to how the provider calculates the diff)

I am not sure if I follow you here @viniciusdc, my problem is not during redeployment but after rebooting.

[krassowski]

This is my k9s when it fails: