Node failed to recover after running out of disk space

[bowenwang1996]

When I try to recover a node after it ran out of disk space, I got

thread 'main' panicked at 'Failed to open the database: DBError(Error { message:
 "Corruption: SST file is ahead of WALs" })', core/store/src/lib.rs:232:23

[ailisp]

This can be seen as a bug of rocksdb but we should not care cases of run out of disk space (many parts of system would out of order once run out of disk). In short why it happens:

1. rocksdb write create a SST file,
2. rocksdb try to dump WAL (write ahead log) into the new SST
3. because of disk out of space 2. was failed

so it appeared to SST ahead of WALs.
I think if delete the largest number SST file, and restart the node would work.

[bowenwang1996]

@ailisp It is fine if the node crashes when it runs out of disk space, but it should not result in an inconsistent state, i.e, we should be able to recover the node.

[ailisp]

So i said the rockdb's bug (it doesn't handle out of disk space properly) But many other code would not work properly once run out of disk. For this specific case you just need to delete the most recent empty sst file and let rocksdb proceed.

…

On Mon, May 11, 2020 at 7:02 PM Bowen Wang ***@***.***> wrote: @ailisp <https://github.com/ailisp> It is fine if the node crashes when it runs out of disk space, but it should not result in an inconsistent state, i.e, we should be able to recover the node. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2625 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADFFFCB5FUOOXAJASDSZX7TRRCU23ANCNFSM4M5NK7UQ> .

[bowenwang1996]

I see. Is it a known issue or should we report it?

[ailisp]

For me it looks like (highlevel, half guessed 😃 )

• rocksdb opens both latest sst and wal file, they both opened successfully, sst file didn't exist before so create new empty file
• write something into sst, failed due to disk space

It depends on how do you think write fail due to run out of space is a normal error situation and should be error handled. IMO it's not, disk space outage should be avoided