RepairDB() drops unflushed non-default column families

[simpkins]

Calling RepairDB() on a valid DB that was shut down cleanly can drop all data in non-default column families if they were not explicitly flushed.

Expected behavior

RepairDB() does not lose data if called on a database that does not actually have any corruption that needs to be repaird.

Actual behavior

RepairDB() drops non-default column families that do not have data that was flushed into an SST.

Steps to reproduce the behavior

Failing test case added here: simpkins@bf36f6d

[ajkr]

@simpkins - There should be a "lost/" directory containing the WALs and MANIFESTs that were deleted by running RepairDB.

[simpkins]

Yes, the data that was dropped is archived into lost/ and can be manually recovered by a human afterwards, but it still seems like a bug that calling RepairDB() on a valid DB removes this data and requires manual recovery afterwards.

Note that this behavior is the same even if you call RepairDB() with an explicit list of all column families that should be present.

[maysamyabandeh]

Thanks @simpkins for the report. I confirm that I can reproduce the issue using the test that you provided.
Apparently the assumption of repair db has been that it would be run only after a disaster so the recovery process has taken the liberty to favor simplicity. It currently discards the manifest, where the column families are registered, and recreate a new manifest by inspecting the existing SST files. The WAL is processed afterwards, when the insertion for new column families are ignored as they are not represented in the recovered manifest.
I am not sure if we should clarify this behavior in the API/docs or spent time to make repair work for this case (and perhaps gets more complicated).

[maysamyabandeh]

I clarified this issue in the wiki: https://github.com/facebook/rocksdb/wiki/RocksDB-Repairer#limitations until it is properly fixed.

[ajkr]

One thing we talked about doing years ago is having a RepairDBRollback(), so users who don't like the effects of RepairDB() can rollback to the previous state without having to manually copy files. This isn't quite as good as preventing RepairDB() from making the DB worse in the first place, but might be a compromise to consider.