[k8s] Lost DB during update

[chevdor]

⚠️ Please verify that this bug has NOT been raised before.

• I checked and didn't find similar issue

🛡️ Security Policy

• I agree to have read this project Security Policy

Description

There are similar issues but close and I just ran into the problem.
Similar issue: #2778

I would have preferred the app to stock and ask than reset everything.

This also leads to a dead end for the user since:

• backup does not work (Deprecated: Since a lot of features were added and this backup feature is a bit unmaintained, it cannot generate or restore a complete backup.)
• Saving data is useless since it gets overwritten

👟 Reproduction steps

Stop old version
Start new version

👀 Expected behavior

The DB is not wiped

😓 Actual Behavior

The DB is wiped/reset

🐻 Uptime-Kuma Version

1.23.1

💻 Operating System and Arch

K8S

🌐 Browser

brave

🐋 Docker Version

K8S 1.27.2

🟩 NodeJS Version

n/a

📝 Relevant log output

No response

[louislam]

K8s is not official supported and most likely you are not pointing to the same volume after update.

[chevdor]

This is not a K8S issue. K8S is just how I run it.
I am pointing 100% to the same folder, my config did not change.
I am also seeing the db in the config location I defined.

I was greeted after the update by the "Let's set your admin password" flow.

[chakflying]

You can post the container logs to help troubleshooting.

[chevdor]

Unfortunately I no longer have logs as I killed the pod to force the update.
I just tried however to kill my current pod, a new one started and did pick everything nicely.

That confirms that my storage config should be fine.
It also points toward the issue being potenitially around upgrades/migrations ?

[chevdor]

noteas that may be helpful:

• I am using only stable release (ie no beta)
• I did jump from 1.22.1 to 1.23.1 without passing thru 1.23.0

[CommanderStorm]

I cannot reproduce this with docker:

• docker stop $(docker ps -qa) && docker rm $(docker ps -qa) && docker volume rm $(docker volume ls -q) (to reset everything, dangerous ^^)
• docker run -it -p 3001:3001 -v uptime-kuma:/app/data louislam/uptime-kuma:1.22.1
• stop said container
• docker run -it -p 3001:3001 -v uptime-kuma:/app/data louislam/uptime-kuma:1.23.1

This leads me to believe that there is something else going on.
How are you running the container exactly (storage-wise, helmchart-wise, k8s-version-wise, ...)

[CommanderStorm]

Another note: How did you upgrade the apps version exactly?
Could it be that you are deploying another verison of the app and tearing down the old version thus deleting the persistant volume you defined?

[chevdor]

Using k8s, if you kill a pod (that was made in a deployment), a new pod gets created as replacement.
Since I am using imagePullPolicy: Always, the freshest image gets pulled in.

So to upgrade, I simply killed the older pod. Storage is handled by k8s and as long as you have storage outside of the container (ie a mounted volume), k8s makes sure the old pod is down before the new one takes over storage.

Could it be that you are deploying another verison of the app and tearing down the old version thus deleting the persistant volume you defined?

This is a fair question but this is not the case and the test I made confirms it is not what is happening. Killing my current pod, does trigger the same process I described, it pulled the latest image (which is still the same version) and starts again.
When I killed the v1.23.1 pod, it was replaced with a new pod and all the data was still available.

[louislam]

Some directions:

1. Use a sqlite gui tool such as SQLite Expert Personal to see what is inside the database kuma.db.
2. Go into the container's terminal and check the size of kuma.db at /app/data.

[chevdor]

Here the sizes already:

[  519050676]  .
├── [    3155380]  sub
│   ├── [      16384]  docker-tls
│   ├── [       3164]  error.log
│   ├── [    1945600]  kuma.db
│   ├── [      32768]  kuma.db-shm
│   ├── [    1108312]  kuma.db-wal
│   ├── [      16384]  screenshots
│   └── [      16384]  upload
├── [  515862528]  subkuma.db
└── [      16384]  subupload

[louislam]

You should be using DATA_DIR

Fixed the data directory issue if you specified a path in DATA_DIR without tailing slash
In case you did this, you may need to manually correct the path like /home/userkuma_data/kuma.db to /home/user/kuma_data/kuma.db)

https://github.com/louislam/uptime-kuma/releases/tag/1.23.0

In your case, you should:

1. rename sub to sub_bak
2. rename subkuma.db to sub/kuma.db
3. rename subupload to sub/upload
4. start the container

[chevdor]

Thanks @louislam, this does explain my issue and thanks to your explanation, I could restore all my previous data.
I was indeed using a DATA_DIR without trailing / and that explains the whole issue.

After adding the trailing /, fixing the folders as mentioned above and killing the pod (which made a new one start), I could get back on track with all the probes I had defined earlier.

❤️