Page storage bug fix #87

JaySon-Huang · 2019-06-21T07:47:07Z

ensure PageFile with meta && data, in case gc drop file but be killed before drop dir
turn PageStorage::Config::sync_on_write = true by default
avoid PageStorage::gc concurrent called by multi-threads
print PageFile's path if checksum is not correct

1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct

flowbehappy · 2019-06-21T08:16:23Z

dbms/src/Storages/Page/PageFile.cpp

                return 0;
+            }
+        }
+        else


It is expected to throw an exception if must_exist == false && errno != ENOENT.

JaySon-Huang · 2019-06-23T08:54:09Z

/run-integration-tests

* add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT

…97) * Page storage bug fix (#87) * add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT * add RefPage && ref counting * 1. move PageCacheMap into isolated file 2. rename PageCacheMap -> PageEntryMap * accept non-exist Page reference/del while being used by WriteBatch or Gc * adjust gc on RefPages * add more test on gc RefPages * add memory usage comparing between vec/hashmap/treemap * fix tests code format * Bug fix: Add RefPage to non exist Page * Add unit test for PageStorage * fix bug * stress test for benchmark * fix bug of PageEntryMap updating RefPage * Keep RefPage entry ref-count if it has been gc and move to another place * MVCC on PageMap, All previse tests passed. TODO: refactor on read_mutex on PageStorage * PageStorage gc: only delete file that are not used by any version * fix bug under multi-threads; remove unused code && format codes * 1. support snapshot read 2. standalone VersionSet/MultiVersionCountable template * 1. Apply merge delta if no read ref 2. Apply generate new delta if has read ref 3. Delta merged when snapshot released 4. Add simple restore test 5. Use template and constexpr instead of copying codes in PageEntryMap * add test for PageStorage snapshot read * use typed test * Bug fix: Add RefPage to non exist Page * fix bug * Delta merged when view is finished * 1. VersionDeltaSet: add snapshots for snapshot linked-list, so that gc can collect valid PageFiles 2. fix some bugs * do compact on delta -> base * rebase after do compact on delta * fix bugs * avoid duplicted codes * 1. new find API 2. all tests passed * 1. remove legacy API 2. remove duplicated codes * reformat * apply inplace if there are no readers * fix bug of applying edits * fix bug of gc apply * fix bug of merging PageEntryMap * fix bug of invalid ref page * fix bug: 1. listAllLiveFiles 2. double compaction on same version * tmp * fixbug * split PageEntryMapView to single cpp file * minor fix * 1. Avoid visit same version multiple time in PageEntryMapDeltaVersionSet::listAllLiveFiles 2. use template to reduce duplicated code for gcApply * todo mark * fix bug: RefPage -> RefPage record may become invalid record when PageStorage GC * remove unused function * simple fix on unittest * fix bug: PageEntryMapView::isRefId * fix bug: PageEntryMapDeltaBuilder::applyPut * fix * fix bug: PageEntryView::validNormalPageIds filter out tombstone of PageEntry * iterator over PageEntryMap * remove unused tests * Refactor on PageEntryMapDeltaVersionSet * Reduce lock range when PageStorage Snapshot is release * Add metrics for PageStorage MVCC * rename some classes * reformat. * use exception instead * adress comment * adress comment * fix compiler error under gcc * fix compiler error under gcc

* add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT

…ingcap#97) * Page storage bug fix (pingcap#87) * add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT * add RefPage && ref counting * 1. move PageCacheMap into isolated file 2. rename PageCacheMap -> PageEntryMap * accept non-exist Page reference/del while being used by WriteBatch or Gc * adjust gc on RefPages * add more test on gc RefPages * add memory usage comparing between vec/hashmap/treemap * fix tests code format * Bug fix: Add RefPage to non exist Page * Add unit test for PageStorage * fix bug * stress test for benchmark * fix bug of PageEntryMap updating RefPage * Keep RefPage entry ref-count if it has been gc and move to another place * MVCC on PageMap, All previse tests passed. TODO: refactor on read_mutex on PageStorage * PageStorage gc: only delete file that are not used by any version * fix bug under multi-threads; remove unused code && format codes * 1. support snapshot read 2. standalone VersionSet/MultiVersionCountable template * 1. Apply merge delta if no read ref 2. Apply generate new delta if has read ref 3. Delta merged when snapshot released 4. Add simple restore test 5. Use template and constexpr instead of copying codes in PageEntryMap * add test for PageStorage snapshot read * use typed test * Bug fix: Add RefPage to non exist Page * fix bug * Delta merged when view is finished * 1. VersionDeltaSet: add snapshots for snapshot linked-list, so that gc can collect valid PageFiles 2. fix some bugs * do compact on delta -> base * rebase after do compact on delta * fix bugs * avoid duplicted codes * 1. new find API 2. all tests passed * 1. remove legacy API 2. remove duplicated codes * reformat * apply inplace if there are no readers * fix bug of applying edits * fix bug of gc apply * fix bug of merging PageEntryMap * fix bug of invalid ref page * fix bug: 1. listAllLiveFiles 2. double compaction on same version * tmp * fixbug * split PageEntryMapView to single cpp file * minor fix * 1. Avoid visit same version multiple time in PageEntryMapDeltaVersionSet::listAllLiveFiles 2. use template to reduce duplicated code for gcApply * todo mark * fix bug: RefPage -> RefPage record may become invalid record when PageStorage GC * remove unused function * simple fix on unittest * fix bug: PageEntryMapView::isRefId * fix bug: PageEntryMapDeltaBuilder::applyPut * fix * fix bug: PageEntryView::validNormalPageIds filter out tombstone of PageEntry * iterator over PageEntryMap * remove unused tests * Refactor on PageEntryMapDeltaVersionSet * Reduce lock range when PageStorage Snapshot is release * Add metrics for PageStorage MVCC * rename some classes * reformat. * use exception instead * adress comment * adress comment * fix compiler error under gcc * fix compiler error under gcc