Sqlite database corruption

[lettore]

I suffered a silent DB Sqlite corruption, silent because there's no sign to understand that Vaultwarden was running with a corrupted database.
I'm using Vaultwarden through Chrome extensions and the Android App, both are connecting to Vaultwarden to keep data in sync, and it's working at least writing, never had a error warning.
Today I logged in with the web-ui just to find my vault empty.
In the container log I see lot of errors:
thread '<unnamed>' panicked at 'Error loading ciphers: DatabaseError(__Unknown, "database disk image is malformed")', src/db/models/cipher.rs:470:41
This SHOULD exit the container immediately, what's the point running a program storing sensitive and important data that is accepting writing to a corrupted database?
Why don't send an email saying that there's a problem with the server?
Why don't raise an alert in the webpage saying that your data is lost?
You can login, add a new entry and nothing tell you about the problem, after you added the new entry you can see it in the page, but after a reload is gone.
I have 2 backups of the DB file, both running every 6 hours, because is very sensitive data.
I found that the only backup working is from 6 months ago.
I'm really lucky that the Chrome extension and the Android app still have all my passwords updated so I can export them and rebuild all.
I kindly ask you to exit the container immediately after a panic error like this, so it's possible to diagnose correctly the problem.

[BlackDex]

I have tried to break the database while running, and i was not able to do that, maybe i need to wait a bit longer or something, but even editing the file with a hex editor didn't made it broken while running. However, when i stopped and started vaultwarden again it didn't started and stopped with an error.

Also, if you say that you were able to store and sync the ciphers between different clients, then up-until that point the database should have been ok. I even was able, after modifying the database file with a hex editor to create a backup via the admin interface.

So, it's hard for me to actually try and debug this. I have kept the server running for a while, i even shredded the wal and shm files etc, and still was able to create a backup, probably because it has some stuff in memory which was helpful in fixing the database i think.

Also, besides all this, I don't know how you create your backups, but if you for example copy only the .sqlite3 file while there are shm and wal files, that will definitely break it. Also, if you somehow copy an old DB over the existing DB, that could cause some strange stuff.

And if you are using a network filesystem, like Samba/CIFS or NFS, that also has issues with sqlite3 files.
In any case, I kinda agree that it should stop somehow, but i would need to be able to mimic this somehow, which i'm not.

I would suggest to read the following wiki articles regarding on how to backup:

• https://github.com/dani-garcia/vaultwarden/wiki/Backing-up-your-vault
• https://github.com/dani-garcia/vaultwarden/wiki/General-%28not-docker%29

Currently I'm afraid we can't really do anything about this until we have some way on reproducing this same kind of behavior.

[lettore]

Database is located on ZFS filesystem, one backup is done with ZFS snapshot and another with the command sqlite3 db.sqlite3 .dump.
I'm quite experienced to know how to backup a database, and not mounting it on NFS or copying it while it's being written.
I was able finally to recover the corrupted file, it had 4 error in ciphers index and 2 in devices. I reindexed the ciphers index and cleared the device table. Then I still had a page error in database so I made a sql dump and recreated the database.
I'm able to read from the corrupted database with SQL tools, so it was not completely corrupted and it makes sense why Vaultwarden was able to open it, using it to check the login password etc. the problem was in the ciphers index, when it needs to load all the ciphers it get into panic. This not happens when it just need to write a new entry, so it's possible that the Chrome extension was sending the new password and Vaultwarden correctly write it, but when asking for changes it gets no reply as Vaultwarden was not able to read correctly the ciphers.
I still have the corrupted database so I have no problem to do tests if you need to understand what's happening.
My request anyway it's really simple, beside how is corrupted the database, Vaultwarden in Docker log throws a panic error. This is an error that anyone can understand as critical, so it should be handled properly. Vaultwarden shouldn't just show an empty page just because is not able to read the ciphers table, it SHOULD report the error in the frontend explaining why it's unable to load the ciphers table, and possibly deny any further write to that database. The best would be also to reply with an error 5xx to any API requests so also the Android app or the Chrome extension will be aware of a problem with the server.

[BlackDex]

In general the .dump is discouraged on running and opened databases. It's much better and safer to use either .backup or VACUUM INTO, since with dump you still need to store the wal and shm as far as i know, and you really really need to be sure that there are no transactions active

Also, same goes for ZFS snapshotting, that isn't instantaneous, and the wal/shm and sqlite3 could get out-off-sync during the snapshot, but most of the time that isn't going to be an issue, but it could.

As for the error catching, it's more easier said then done tracking these specific errors and acting correctly upon those messages. And, it would still be easier if we were able to reproduce something like this to be sure it could be tackled.

Also, you are running a very old version of Vaultwarden 1.24.0 which is from January this year. In the mean time there have been several large updates done on whole Vaultwarden codebase including async functions, database handling and connections, library updates, newer rust versions which could have bugs fixed, base container image etc.

So, is hard to tell for me if this issue is still valid using the latest version available. And it would be better if you are able to provide a more detailed log during an issue like this, so with LOG_LEVEL=trace and RUST_BACKTRACE=full for example, and then the whole backtrace and a few lines before and after, instead of just one line (but, only using the latest stable or testing version, not an older one).

[lettore]

As said, I made a backup every 6 hours and I save a new password no more twice for week, so it's quite difficult to snapshot the database while writing. I have 360 snapshots of the database all with the same problem. The original file on disk is corrupt, then all the backups of 6 months were corrupted in the same way, 6 wrong index. So the backup process was perfect in making a exact copy of an unfortunate corrupted file.
Not sure why you say I'm running version 1.24.0 while I always used the latest Docker tag, I can see ver 2.25.0 in UI.
I'm not trying to diagnose why the database became corrupted. I'm just kindly ask to handle a panic error in a different way. I will add to my backup process a PRAGMA INTEGRITY CHECK on the output just to be sure to have a valid backup file and be notified otherwise.

[BlackDex]

I'm just trying to help produce better backups and prevent a new corrupted database file, using .dump could cause a corruption in some rare cases, .backup and VACUUM INTOare preferred.
And while you maybe are not adding a new password there are some background jobs active which could be doing database stuff on that same moment.

The reason i say you are running 1.24.0 is because of the error message you gave. That specific line only matches 1.24.0 or older.

If this was an old log line from before you updated, then i can understand it, but if this log line is at the end of the log then there is something wrong with your running docker container also.

You mentioned that you still have the old database which is corrupted. I kindly asked if you could provide some more logs including the whole backtrace and a few lines (10 or 20) before and after the whole backtrace with the modified environment variables and with the latest version 1.26.0 or testing so that we have more details using the up-to-date version to try and help you and all other people who might encounter something like this.

And even with that, it's going to be difficult for us to really tackle and test this, since we can't reproduce this easily and make sure the application is doing what it should be in this specific case.

[lettore]

My fault, I was using an old image 1.23.1. Using the latest 1.26.0 it goes not beyond trying to do the migration script, so no way to see how the new version deals a corrupted database.
This is the log file while running the old version, as you can see after the error the server continues to reply with 200 OK to other requests.
mws3f3yy92ndwo9zo3bmffjcq_logs.txt
In Admin page I can see users with the correct number of entries and organizations.
As the new image is not able to migrate the database and exits I can't say if the issue is still present.

[BlackDex]

Thanks for the logs, hopefully there is something useful we can get out of it.

Also as a side-note. The 1.23.1 version was also using an older SQLite library 3.31.0 and the current stable uses 3.35.4, with in between a lot of changes, bug fixes, optimizations etc... Which could also in turn help, or would have helped preventing the file from getting corrupt.

The current testing branch is even using the a very new version 3.39.2.

[mhzawadi]

I too am getting DB corruption, have had it over night. I have hourly backups setup so can normally see when it has issues.

Will grab logs next time and post here, as I have only just found this and have already got things working

[BlackDex]

Besides logs, we also need to know the version of Vaultwarden etc..

But besides all that, Vaultwarden isn't corrupting the database. It is mostly using a wrong way of backing up the database, depleted storage, memory issues, or sudden shutdown etc...

[mhzawadi]

might have found my issue 🤦🏼 , I use bruceforce/vaultwarden-backup as the backup container. That was setup to delete files older then 3 days, as in delete * older than 3 days.

2022-10-23 19:05:01 - INFO - Deleted backup /data/db.sqlite3 after 3 days

my bad

[BlackDex]

Ow my, that is very bad.

[androidacy-user]

We just lost our entire vaultwarden instance because it decided the sqlite database was corrupt overnight and the database was not recoverable through any means i could find. Database is only ever touched by the docker instance itself so can rule that out.

[BlackDex]

Vaultwarden doesn't corrupt a database either. So that is also something to rule out. Most other items could be backups, filesystem issues, memory issues and some other items.

Maybe there is a bit more info in the logs? Also, which version of Vaultwarden were you using.

[lettore]

We just lost our entire vaultwarden instance because it decided the sqlite database was corrupt overnight and the database was not recoverable through any means i could find. Database is only ever touched by the docker instance itself so can rule that out.

Trying recovery mine I saw that it was not corrupted totally, it's just Vaultwarden that refuses to open it. It should be added a tool inside the admin UI to export a corrupted database. With a Windows MySql utility I was able to see all the database correctly, only the last 3 ciphers entries were missing. Try to diagnose where the db is corrupted, as mine had lot of errors in devices table that can be safely deleted.

[androidacy-user]

We couldn't access ours with SQLite3 either.

@BlackDex quite frankly no idea as we restored to a known good point after digging through our backups to find one that also had Vaultwarden (it's not included in our daily backups as it's low priority and not expected to randomly fail). You'll understand our first priority was getting back our data.

[ranasats]

I wanted to share that I just experienced this. I was able to successfully recover the database using the following sqlite3 commands:

Error:

[2025-01-21 16:14:55.591][error][ERROR] Error saving device.
[CAUSE] DatabaseError(
    Unknown,
    "database disk image is malformed",
)

Fix:

1. Shutdown Vaultwarden docker.
2. sqlite3 version 3.46.1: Execute:

sqlite3 db.sqlite3 ".recover" | sqlite3 "recovered.db"

3. rename db.sqlite3 to db.sqlite3.malformed
4. rename recovered.db to db.sqlite3
5. Restart Bitwarden
6. Relogin to app/website
7. create a dummy entry. I did both login and note. No more malformed database error.

I would just like to say, there should 100% be a check while the image starts that shuts down the image if the database is corrupt like this. I am not sure why the entire app is allowed to function as if it's working, and only causing issues if you try to add logins. ALSO this should not be a "low priority" issue at all. In fact it feels like a high-priority issue.

[BlackDex]

I wouldn't remove db.sqlite3 but rename it to old or something to still have a backup of the broken file.

[ranasats]

I wouldn't remove db.sqlite3 but rename it to old or something to still have a backup of the broken file.

Yeah, I agree. I have hourly backups so I wasn't too worried about losing data. One of them would have been fine. I would have went into those if this didn't work the first time around. Edited initial post.

[tessus]

I know I am late to this thread, but I want to make it perfectly clear that running sqlite on NFS, SAMBA, or ZFS will not work. I don't know if that is on the wiki already but if not, it's a good idea to add it.

[ranasats]

My systems on btfrs, while I haven't used the other two for production, zfs and sqlite should work, as long as backups/snapshots are done while the docker is shut down (best practice for any backup solution though...). Unless you're thinking of some other issue aside from WAL and snapshots happening at the same time.

[tessus]

e.g. openzfs/zfs#14290

Btw, btrfs also has an issue with fsync being delayed during automatic snapshot creation and deletion.