concurrent flush cause data loss after GC #93
Assignees
Comments
This was referenced Oct 15, 2019
yiwu-arbug
pushed a commit
to tikv/rocksdb
that referenced
this issue
Oct 16, 2019
Move `WaitForFlushMemTable` API to public. It is needed for hotfix of tikv/titan#93 Signed-off-by: Yi Wu <yiwu@pingcap.com>
yiwu-arbug
pushed a commit
to tikv/rocksdb
that referenced
this issue
Oct 16, 2019
Move `WaitForFlushMemTable` API to public. It is needed for hotfix of tikv/titan#93 Signed-off-by: Yi Wu <yiwu@pingcap.com>
yiwu-arbug
pushed a commit
that referenced
this issue
Oct 16, 2019
Summary: Temporary fix for #93. Fixing it by waiting until no flush is ongoing before starting a GC job. The drawback is GC can starve if flush runs so frequent that there's no gap between flush jobs. Test Plan: See the new test. The test will fail with "Corruption: Missing blob file" error before the fix.
yiwu-arbug
pushed a commit
to yiwu-arbug/titan
that referenced
this issue
Oct 16, 2019
Summary: Temporary fix for tikv#93. Fixing it by waiting until no flush is ongoing before starting a GC job. The drawback is GC can starve if flush runs so frequent that there's no gap between flush jobs. Test Plan: See the new test. The test will fail with "Corruption: Missing blob file" error before the fix.
tabokie
pushed a commit
to tabokie/rocksdb
that referenced
this issue
May 11, 2022
Move `WaitForFlushMemTable` API to public. It is needed for hotfix of tikv/titan#93 Signed-off-by: Yi Wu <yiwu@pingcap.com> Signed-off-by: tabokie <xy.tao@outlook.com>
tabokie
pushed a commit
to tikv/rocksdb
that referenced
this issue
May 12, 2022
Move `WaitForFlushMemTable` API to public. It is needed for hotfix of tikv/titan#93 Signed-off-by: Yi Wu <yiwu@pingcap.com> Signed-off-by: tabokie <xy.tao@outlook.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Titan use event listener to hook
OnFlushCompleted. When the event triggered, Titan assume corresponding memtable has been converted to SST, and mark blob file generated from the flush as normal file, so that GC can pick the file up. However, the assumption is not correct. If there are concurrent flush happened,OnFlushCompletedcan fire before memtable being converted to SST (facebook/rocksdb#5892). If GC pick up the blob file before memtable flush finished, it can receive is_blob_index=false for all keys in the blob file (since they are still in memtable), and drop all data in the file by mistake, causing data loss.Reproduced with unit test: yiwu-arbug@5172201
The text was updated successfully, but these errors were encountered: