[FLASH-318/319/343/344] PageStorage: RefPages, MVCC && Snapshot read #97
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/run-integration-tests |
JaySon-Huang
changed the title
JaySon-Huang
force-pushed
the
jayson/PS_RefPages_MVCC
branch
2 times, most recently
from
caae1e1 to
b0891ed
Compare
* add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT
2. rename PageCacheMap -> PageEntryMap
…ex on PageStorage
remove unused code && format codes
2. standalone VersionSet/MultiVersionCountable template
2. Apply generate new delta if has read ref 3. Delta merged when snapshot released 4. Add simple restore test 5. Use template and constexpr instead of copying codes in PageEntryMap
flowbehappy
reviewed
Aug 1, 2019
flowbehappy
reviewed
Aug 1, 2019
|
|
||
| void DeltaVersionEditAcceptor::applyPut(PageEntriesEdit::EditRecord & rec) | ||
| { | ||
| assert(rec.type == WriteBatch::WriteType::PUT); |
There was a problem hiding this comment.
Normally we don't use assert, use if(unlikely(...)){throw XXXException} instead.
There was a problem hiding this comment.
Sometime I use assert to tell developer(myself or others) that they should ensure that situation. And developer should know that they use that function incorrectly and fix that in coding.
I will use Exception instead of assert at these codes
- https://github.com/pingcap/tics/pull/97/files#diff-595e370142003a9cb3b009da6ac2d1e8R284
- https://github.com/pingcap/tics/pull/97/files#diff-5087178bb0477713223fcbbd41649aefR185
https://github.com/pingcap/tics/pull/97/files#diff-01011459ba99e2477b3fb0c9faa8c04cR68
There was a problem hiding this comment.
destructor has a implicit non-throwing exception specification
Compiler don't allow to throw exception at destructor function
|
/build |
flowbehappy
reviewed
Aug 2, 2019
|
/run-integration-tests |
1 similar comment
|
/run-integration-tests |
JaySon-Huang
added a commit
to JaySon-Huang/tiflash
that referenced
this pull request
Aug 26, 2019
…ingcap#97) * Page storage bug fix (pingcap#87) * add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT * add RefPage && ref counting * 1. move PageCacheMap into isolated file 2. rename PageCacheMap -> PageEntryMap * accept non-exist Page reference/del while being used by WriteBatch or Gc * adjust gc on RefPages * add more test on gc RefPages * add memory usage comparing between vec/hashmap/treemap * fix tests code format * Bug fix: Add RefPage to non exist Page * Add unit test for PageStorage * fix bug * stress test for benchmark * fix bug of PageEntryMap updating RefPage * Keep RefPage entry ref-count if it has been gc and move to another place * MVCC on PageMap, All previse tests passed. TODO: refactor on read_mutex on PageStorage * PageStorage gc: only delete file that are not used by any version * fix bug under multi-threads; remove unused code && format codes * 1. support snapshot read 2. standalone VersionSet/MultiVersionCountable template * 1. Apply merge delta if no read ref 2. Apply generate new delta if has read ref 3. Delta merged when snapshot released 4. Add simple restore test 5. Use template and constexpr instead of copying codes in PageEntryMap * add test for PageStorage snapshot read * use typed test * Bug fix: Add RefPage to non exist Page * fix bug * Delta merged when view is finished * 1. VersionDeltaSet: add snapshots for snapshot linked-list, so that gc can collect valid PageFiles 2. fix some bugs * do compact on delta -> base * rebase after do compact on delta * fix bugs * avoid duplicted codes * 1. new find API 2. all tests passed * 1. remove legacy API 2. remove duplicated codes * reformat * apply inplace if there are no readers * fix bug of applying edits * fix bug of gc apply * fix bug of merging PageEntryMap * fix bug of invalid ref page * fix bug: 1. listAllLiveFiles 2. double compaction on same version * tmp * fixbug * split PageEntryMapView to single cpp file * minor fix * 1. Avoid visit same version multiple time in PageEntryMapDeltaVersionSet::listAllLiveFiles 2. use template to reduce duplicated code for gcApply * todo mark * fix bug: RefPage -> RefPage record may become invalid record when PageStorage GC * remove unused function * simple fix on unittest * fix bug: PageEntryMapView::isRefId * fix bug: PageEntryMapDeltaBuilder::applyPut * fix * fix bug: PageEntryView::validNormalPageIds filter out tombstone of PageEntry * iterator over PageEntryMap * remove unused tests * Refactor on PageEntryMapDeltaVersionSet * Reduce lock range when PageStorage Snapshot is release * Add metrics for PageStorage MVCC * rename some classes * reformat. * use exception instead * adress comment * adress comment * fix compiler error under gcc * fix compiler error under gcc
JaySon-Huang
added a commit
to JaySon-Huang/tiflash
that referenced
this pull request
Sep 5, 2019
…ingcap#97) * Page storage bug fix (pingcap#87) * add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT * add RefPage && ref counting * 1. move PageCacheMap into isolated file 2. rename PageCacheMap -> PageEntryMap * accept non-exist Page reference/del while being used by WriteBatch or Gc * adjust gc on RefPages * add more test on gc RefPages * add memory usage comparing between vec/hashmap/treemap * fix tests code format * Bug fix: Add RefPage to non exist Page * Add unit test for PageStorage * fix bug * stress test for benchmark * fix bug of PageEntryMap updating RefPage * Keep RefPage entry ref-count if it has been gc and move to another place * MVCC on PageMap, All previse tests passed. TODO: refactor on read_mutex on PageStorage * PageStorage gc: only delete file that are not used by any version * fix bug under multi-threads; remove unused code && format codes * 1. support snapshot read 2. standalone VersionSet/MultiVersionCountable template * 1. Apply merge delta if no read ref 2. Apply generate new delta if has read ref 3. Delta merged when snapshot released 4. Add simple restore test 5. Use template and constexpr instead of copying codes in PageEntryMap * add test for PageStorage snapshot read * use typed test * Bug fix: Add RefPage to non exist Page * fix bug * Delta merged when view is finished * 1. VersionDeltaSet: add snapshots for snapshot linked-list, so that gc can collect valid PageFiles 2. fix some bugs * do compact on delta -> base * rebase after do compact on delta * fix bugs * avoid duplicted codes * 1. new find API 2. all tests passed * 1. remove legacy API 2. remove duplicated codes * reformat * apply inplace if there are no readers * fix bug of applying edits * fix bug of gc apply * fix bug of merging PageEntryMap * fix bug of invalid ref page * fix bug: 1. listAllLiveFiles 2. double compaction on same version * tmp * fixbug * split PageEntryMapView to single cpp file * minor fix * 1. Avoid visit same version multiple time in PageEntryMapDeltaVersionSet::listAllLiveFiles 2. use template to reduce duplicated code for gcApply * todo mark * fix bug: RefPage -> RefPage record may become invalid record when PageStorage GC * remove unused function * simple fix on unittest * fix bug: PageEntryMapView::isRefId * fix bug: PageEntryMapDeltaBuilder::applyPut * fix * fix bug: PageEntryView::validNormalPageIds filter out tombstone of PageEntry * iterator over PageEntryMap * remove unused tests * Refactor on PageEntryMapDeltaVersionSet * Reduce lock range when PageStorage Snapshot is release * Add metrics for PageStorage MVCC * rename some classes * reformat. * use exception instead * adress comment * adress comment * fix compiler error under gcc * fix compiler error under gcc
JaySon-Huang
added a commit
to JaySon-Huang/tiflash
that referenced
this pull request
Sep 12, 2019
…ingcap#97) * Page storage bug fix (pingcap#87) * add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT * add RefPage && ref counting * 1. move PageCacheMap into isolated file 2. rename PageCacheMap -> PageEntryMap * accept non-exist Page reference/del while being used by WriteBatch or Gc * adjust gc on RefPages * add more test on gc RefPages * add memory usage comparing between vec/hashmap/treemap * fix tests code format * Bug fix: Add RefPage to non exist Page * Add unit test for PageStorage * fix bug * stress test for benchmark * fix bug of PageEntryMap updating RefPage * Keep RefPage entry ref-count if it has been gc and move to another place * MVCC on PageMap, All previse tests passed. TODO: refactor on read_mutex on PageStorage * PageStorage gc: only delete file that are not used by any version * fix bug under multi-threads; remove unused code && format codes * 1. support snapshot read 2. standalone VersionSet/MultiVersionCountable template * 1. Apply merge delta if no read ref 2. Apply generate new delta if has read ref 3. Delta merged when snapshot released 4. Add simple restore test 5. Use template and constexpr instead of copying codes in PageEntryMap * add test for PageStorage snapshot read * use typed test * Bug fix: Add RefPage to non exist Page * fix bug * Delta merged when view is finished * 1. VersionDeltaSet: add snapshots for snapshot linked-list, so that gc can collect valid PageFiles 2. fix some bugs * do compact on delta -> base * rebase after do compact on delta * fix bugs * avoid duplicted codes * 1. new find API 2. all tests passed * 1. remove legacy API 2. remove duplicated codes * reformat * apply inplace if there are no readers * fix bug of applying edits * fix bug of gc apply * fix bug of merging PageEntryMap * fix bug of invalid ref page * fix bug: 1. listAllLiveFiles 2. double compaction on same version * tmp * fixbug * split PageEntryMapView to single cpp file * minor fix * 1. Avoid visit same version multiple time in PageEntryMapDeltaVersionSet::listAllLiveFiles 2. use template to reduce duplicated code for gcApply * todo mark * fix bug: RefPage -> RefPage record may become invalid record when PageStorage GC * remove unused function * simple fix on unittest * fix bug: PageEntryMapView::isRefId * fix bug: PageEntryMapDeltaBuilder::applyPut * fix * fix bug: PageEntryView::validNormalPageIds filter out tombstone of PageEntry * iterator over PageEntryMap * remove unused tests * Refactor on PageEntryMapDeltaVersionSet * Reduce lock range when PageStorage Snapshot is release * Add metrics for PageStorage MVCC * rename some classes * reformat. * use exception instead * adress comment * adress comment * fix compiler error under gcc * fix compiler error under gcc
JaySon-Huang
added a commit
to JaySon-Huang/tiflash
that referenced
this pull request
Sep 18, 2019
…ingcap#97) * Page storage bug fix (pingcap#87) * add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT * add RefPage && ref counting * 1. move PageCacheMap into isolated file 2. rename PageCacheMap -> PageEntryMap * accept non-exist Page reference/del while being used by WriteBatch or Gc * adjust gc on RefPages * add more test on gc RefPages * add memory usage comparing between vec/hashmap/treemap * fix tests code format * Bug fix: Add RefPage to non exist Page * Add unit test for PageStorage * fix bug * stress test for benchmark * fix bug of PageEntryMap updating RefPage * Keep RefPage entry ref-count if it has been gc and move to another place * MVCC on PageMap, All previse tests passed. TODO: refactor on read_mutex on PageStorage * PageStorage gc: only delete file that are not used by any version * fix bug under multi-threads; remove unused code && format codes * 1. support snapshot read 2. standalone VersionSet/MultiVersionCountable template * 1. Apply merge delta if no read ref 2. Apply generate new delta if has read ref 3. Delta merged when snapshot released 4. Add simple restore test 5. Use template and constexpr instead of copying codes in PageEntryMap * add test for PageStorage snapshot read * use typed test * Bug fix: Add RefPage to non exist Page * fix bug * Delta merged when view is finished * 1. VersionDeltaSet: add snapshots for snapshot linked-list, so that gc can collect valid PageFiles 2. fix some bugs * do compact on delta -> base * rebase after do compact on delta * fix bugs * avoid duplicted codes * 1. new find API 2. all tests passed * 1. remove legacy API 2. remove duplicated codes * reformat * apply inplace if there are no readers * fix bug of applying edits * fix bug of gc apply * fix bug of merging PageEntryMap * fix bug of invalid ref page * fix bug: 1. listAllLiveFiles 2. double compaction on same version * tmp * fixbug * split PageEntryMapView to single cpp file * minor fix * 1. Avoid visit same version multiple time in PageEntryMapDeltaVersionSet::listAllLiveFiles 2. use template to reduce duplicated code for gcApply * todo mark * fix bug: RefPage -> RefPage record may become invalid record when PageStorage GC * remove unused function * simple fix on unittest * fix bug: PageEntryMapView::isRefId * fix bug: PageEntryMapDeltaBuilder::applyPut * fix * fix bug: PageEntryView::validNormalPageIds filter out tombstone of PageEntry * iterator over PageEntryMap * remove unused tests * Refactor on PageEntryMapDeltaVersionSet * Reduce lock range when PageStorage Snapshot is release * Add metrics for PageStorage MVCC * rename some classes * reformat. * use exception instead * adress comment * adress comment * fix compiler error under gcc * fix compiler error under gcc
JaySon-Huang
added a commit
to JaySon-Huang/tiflash
that referenced
this pull request
Sep 23, 2019
…ingcap#97) * Page storage bug fix (pingcap#87) * add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT * add RefPage && ref counting * 1. move PageCacheMap into isolated file 2. rename PageCacheMap -> PageEntryMap * accept non-exist Page reference/del while being used by WriteBatch or Gc * adjust gc on RefPages * add more test on gc RefPages * add memory usage comparing between vec/hashmap/treemap * fix tests code format * Bug fix: Add RefPage to non exist Page * Add unit test for PageStorage * fix bug * stress test for benchmark * fix bug of PageEntryMap updating RefPage * Keep RefPage entry ref-count if it has been gc and move to another place * MVCC on PageMap, All previse tests passed. TODO: refactor on read_mutex on PageStorage * PageStorage gc: only delete file that are not used by any version * fix bug under multi-threads; remove unused code && format codes * 1. support snapshot read 2. standalone VersionSet/MultiVersionCountable template * 1. Apply merge delta if no read ref 2. Apply generate new delta if has read ref 3. Delta merged when snapshot released 4. Add simple restore test 5. Use template and constexpr instead of copying codes in PageEntryMap * add test for PageStorage snapshot read * use typed test * Bug fix: Add RefPage to non exist Page * fix bug * Delta merged when view is finished * 1. VersionDeltaSet: add snapshots for snapshot linked-list, so that gc can collect valid PageFiles 2. fix some bugs * do compact on delta -> base * rebase after do compact on delta * fix bugs * avoid duplicted codes * 1. new find API 2. all tests passed * 1. remove legacy API 2. remove duplicated codes * reformat * apply inplace if there are no readers * fix bug of applying edits * fix bug of gc apply * fix bug of merging PageEntryMap * fix bug of invalid ref page * fix bug: 1. listAllLiveFiles 2. double compaction on same version * tmp * fixbug * split PageEntryMapView to single cpp file * minor fix * 1. Avoid visit same version multiple time in PageEntryMapDeltaVersionSet::listAllLiveFiles 2. use template to reduce duplicated code for gcApply * todo mark * fix bug: RefPage -> RefPage record may become invalid record when PageStorage GC * remove unused function * simple fix on unittest * fix bug: PageEntryMapView::isRefId * fix bug: PageEntryMapDeltaBuilder::applyPut * fix * fix bug: PageEntryView::validNormalPageIds filter out tombstone of PageEntry * iterator over PageEntryMap * remove unused tests * Refactor on PageEntryMapDeltaVersionSet * Reduce lock range when PageStorage Snapshot is release * Add metrics for PageStorage MVCC * rename some classes * reformat. * use exception instead * adress comment * adress comment * fix compiler error under gcc * fix compiler error under gcc
JaySon-Huang
added a commit
that referenced
this pull request
Sep 26, 2019
…97) * Page storage bug fix (#87) * add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT * add RefPage && ref counting * 1. move PageCacheMap into isolated file 2. rename PageCacheMap -> PageEntryMap * accept non-exist Page reference/del while being used by WriteBatch or Gc * adjust gc on RefPages * add more test on gc RefPages * add memory usage comparing between vec/hashmap/treemap * fix tests code format * Bug fix: Add RefPage to non exist Page * Add unit test for PageStorage * fix bug * stress test for benchmark * fix bug of PageEntryMap updating RefPage * Keep RefPage entry ref-count if it has been gc and move to another place * MVCC on PageMap, All previse tests passed. TODO: refactor on read_mutex on PageStorage * PageStorage gc: only delete file that are not used by any version * fix bug under multi-threads; remove unused code && format codes * 1. support snapshot read 2. standalone VersionSet/MultiVersionCountable template * 1. Apply merge delta if no read ref 2. Apply generate new delta if has read ref 3. Delta merged when snapshot released 4. Add simple restore test 5. Use template and constexpr instead of copying codes in PageEntryMap * add test for PageStorage snapshot read * use typed test * Bug fix: Add RefPage to non exist Page * fix bug * Delta merged when view is finished * 1. VersionDeltaSet: add snapshots for snapshot linked-list, so that gc can collect valid PageFiles 2. fix some bugs * do compact on delta -> base * rebase after do compact on delta * fix bugs * avoid duplicted codes * 1. new find API 2. all tests passed * 1. remove legacy API 2. remove duplicated codes * reformat * apply inplace if there are no readers * fix bug of applying edits * fix bug of gc apply * fix bug of merging PageEntryMap * fix bug of invalid ref page * fix bug: 1. listAllLiveFiles 2. double compaction on same version * tmp * fixbug * split PageEntryMapView to single cpp file * minor fix * 1. Avoid visit same version multiple time in PageEntryMapDeltaVersionSet::listAllLiveFiles 2. use template to reduce duplicated code for gcApply * todo mark * fix bug: RefPage -> RefPage record may become invalid record when PageStorage GC * remove unused function * simple fix on unittest * fix bug: PageEntryMapView::isRefId * fix bug: PageEntryMapDeltaBuilder::applyPut * fix * fix bug: PageEntryView::validNormalPageIds filter out tombstone of PageEntry * iterator over PageEntryMap * remove unused tests * Refactor on PageEntryMapDeltaVersionSet * Reduce lock range when PageStorage Snapshot is release * Add metrics for PageStorage MVCC * rename some classes * reformat. * use exception instead * adress comment * adress comment * fix compiler error under gcc * fix compiler error under gcc
JaySon-Huang
added a commit
to JaySon-Huang/tiflash
that referenced
this pull request
Oct 17, 2019
…ingcap#97) * Page storage bug fix (pingcap#87) * add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT * add RefPage && ref counting * 1. move PageCacheMap into isolated file 2. rename PageCacheMap -> PageEntryMap * accept non-exist Page reference/del while being used by WriteBatch or Gc * adjust gc on RefPages * add more test on gc RefPages * add memory usage comparing between vec/hashmap/treemap * fix tests code format * Bug fix: Add RefPage to non exist Page * Add unit test for PageStorage * fix bug * stress test for benchmark * fix bug of PageEntryMap updating RefPage * Keep RefPage entry ref-count if it has been gc and move to another place * MVCC on PageMap, All previse tests passed. TODO: refactor on read_mutex on PageStorage * PageStorage gc: only delete file that are not used by any version * fix bug under multi-threads; remove unused code && format codes * 1. support snapshot read 2. standalone VersionSet/MultiVersionCountable template * 1. Apply merge delta if no read ref 2. Apply generate new delta if has read ref 3. Delta merged when snapshot released 4. Add simple restore test 5. Use template and constexpr instead of copying codes in PageEntryMap * add test for PageStorage snapshot read * use typed test * Bug fix: Add RefPage to non exist Page * fix bug * Delta merged when view is finished * 1. VersionDeltaSet: add snapshots for snapshot linked-list, so that gc can collect valid PageFiles 2. fix some bugs * do compact on delta -> base * rebase after do compact on delta * fix bugs * avoid duplicted codes * 1. new find API 2. all tests passed * 1. remove legacy API 2. remove duplicated codes * reformat * apply inplace if there are no readers * fix bug of applying edits * fix bug of gc apply * fix bug of merging PageEntryMap * fix bug of invalid ref page * fix bug: 1. listAllLiveFiles 2. double compaction on same version * tmp * fixbug * split PageEntryMapView to single cpp file * minor fix * 1. Avoid visit same version multiple time in PageEntryMapDeltaVersionSet::listAllLiveFiles 2. use template to reduce duplicated code for gcApply * todo mark * fix bug: RefPage -> RefPage record may become invalid record when PageStorage GC * remove unused function * simple fix on unittest * fix bug: PageEntryMapView::isRefId * fix bug: PageEntryMapDeltaBuilder::applyPut * fix * fix bug: PageEntryView::validNormalPageIds filter out tombstone of PageEntry * iterator over PageEntryMap * remove unused tests * Refactor on PageEntryMapDeltaVersionSet * Reduce lock range when PageStorage Snapshot is release * Add metrics for PageStorage MVCC * rename some classes * reformat. * use exception instead * adress comment * adress comment * fix compiler error under gcc * fix compiler error under gcc
JaySon-Huang
added a commit
to JaySon-Huang/tiflash
that referenced
this pull request
Oct 18, 2019
…ingcap#97) * Page storage bug fix (pingcap#87) * add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT * add RefPage && ref counting * 1. move PageCacheMap into isolated file 2. rename PageCacheMap -> PageEntryMap * accept non-exist Page reference/del while being used by WriteBatch or Gc * adjust gc on RefPages * add more test on gc RefPages * add memory usage comparing between vec/hashmap/treemap * fix tests code format * Bug fix: Add RefPage to non exist Page * Add unit test for PageStorage * fix bug * stress test for benchmark * fix bug of PageEntryMap updating RefPage * Keep RefPage entry ref-count if it has been gc and move to another place * MVCC on PageMap, All previse tests passed. TODO: refactor on read_mutex on PageStorage * PageStorage gc: only delete file that are not used by any version * fix bug under multi-threads; remove unused code && format codes * 1. support snapshot read 2. standalone VersionSet/MultiVersionCountable template * 1. Apply merge delta if no read ref 2. Apply generate new delta if has read ref 3. Delta merged when snapshot released 4. Add simple restore test 5. Use template and constexpr instead of copying codes in PageEntryMap * add test for PageStorage snapshot read * use typed test * Bug fix: Add RefPage to non exist Page * fix bug * Delta merged when view is finished * 1. VersionDeltaSet: add snapshots for snapshot linked-list, so that gc can collect valid PageFiles 2. fix some bugs * do compact on delta -> base * rebase after do compact on delta * fix bugs * avoid duplicted codes * 1. new find API 2. all tests passed * 1. remove legacy API 2. remove duplicated codes * reformat * apply inplace if there are no readers * fix bug of applying edits * fix bug of gc apply * fix bug of merging PageEntryMap * fix bug of invalid ref page * fix bug: 1. listAllLiveFiles 2. double compaction on same version * tmp * fixbug * split PageEntryMapView to single cpp file * minor fix * 1. Avoid visit same version multiple time in PageEntryMapDeltaVersionSet::listAllLiveFiles 2. use template to reduce duplicated code for gcApply * todo mark * fix bug: RefPage -> RefPage record may become invalid record when PageStorage GC * remove unused function * simple fix on unittest * fix bug: PageEntryMapView::isRefId * fix bug: PageEntryMapDeltaBuilder::applyPut * fix * fix bug: PageEntryView::validNormalPageIds filter out tombstone of PageEntry * iterator over PageEntryMap * remove unused tests * Refactor on PageEntryMapDeltaVersionSet * Reduce lock range when PageStorage Snapshot is release * Add metrics for PageStorage MVCC * rename some classes * reformat. * use exception instead * adress comment * adress comment * fix compiler error under gcc * fix compiler error under gcc
JaySon-Huang
added a commit
that referenced
this pull request
Oct 22, 2019
…97) * Page storage bug fix (#87) * add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT * add RefPage && ref counting * 1. move PageCacheMap into isolated file 2. rename PageCacheMap -> PageEntryMap * accept non-exist Page reference/del while being used by WriteBatch or Gc * adjust gc on RefPages * add more test on gc RefPages * add memory usage comparing between vec/hashmap/treemap * fix tests code format * Bug fix: Add RefPage to non exist Page * Add unit test for PageStorage * fix bug * stress test for benchmark * fix bug of PageEntryMap updating RefPage * Keep RefPage entry ref-count if it has been gc and move to another place * MVCC on PageMap, All previse tests passed. TODO: refactor on read_mutex on PageStorage * PageStorage gc: only delete file that are not used by any version * fix bug under multi-threads; remove unused code && format codes * 1. support snapshot read 2. standalone VersionSet/MultiVersionCountable template * 1. Apply merge delta if no read ref 2. Apply generate new delta if has read ref 3. Delta merged when snapshot released 4. Add simple restore test 5. Use template and constexpr instead of copying codes in PageEntryMap * add test for PageStorage snapshot read * use typed test * Bug fix: Add RefPage to non exist Page * fix bug * Delta merged when view is finished * 1. VersionDeltaSet: add snapshots for snapshot linked-list, so that gc can collect valid PageFiles 2. fix some bugs * do compact on delta -> base * rebase after do compact on delta * fix bugs * avoid duplicted codes * 1. new find API 2. all tests passed * 1. remove legacy API 2. remove duplicated codes * reformat * apply inplace if there are no readers * fix bug of applying edits * fix bug of gc apply * fix bug of merging PageEntryMap * fix bug of invalid ref page * fix bug: 1. listAllLiveFiles 2. double compaction on same version * tmp * fixbug * split PageEntryMapView to single cpp file * minor fix * 1. Avoid visit same version multiple time in PageEntryMapDeltaVersionSet::listAllLiveFiles 2. use template to reduce duplicated code for gcApply * todo mark * fix bug: RefPage -> RefPage record may become invalid record when PageStorage GC * remove unused function * simple fix on unittest * fix bug: PageEntryMapView::isRefId * fix bug: PageEntryMapDeltaBuilder::applyPut * fix * fix bug: PageEntryView::validNormalPageIds filter out tombstone of PageEntry * iterator over PageEntryMap * remove unused tests * Refactor on PageEntryMapDeltaVersionSet * Reduce lock range when PageStorage Snapshot is release * Add metrics for PageStorage MVCC * rename some classes * reformat. * use exception instead * adress comment * adress comment * fix compiler error under gcc * fix compiler error under gcc
JaySon-Huang
added a commit
to JaySon-Huang/tiflash
that referenced
this pull request
Oct 23, 2019
…ingcap#97) * Page storage bug fix (pingcap#87) * add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT * add RefPage && ref counting * 1. move PageCacheMap into isolated file 2. rename PageCacheMap -> PageEntryMap * accept non-exist Page reference/del while being used by WriteBatch or Gc * adjust gc on RefPages * add more test on gc RefPages * add memory usage comparing between vec/hashmap/treemap * fix tests code format * Bug fix: Add RefPage to non exist Page * Add unit test for PageStorage * fix bug * stress test for benchmark * fix bug of PageEntryMap updating RefPage * Keep RefPage entry ref-count if it has been gc and move to another place * MVCC on PageMap, All previse tests passed. TODO: refactor on read_mutex on PageStorage * PageStorage gc: only delete file that are not used by any version * fix bug under multi-threads; remove unused code && format codes * 1. support snapshot read 2. standalone VersionSet/MultiVersionCountable template * 1. Apply merge delta if no read ref 2. Apply generate new delta if has read ref 3. Delta merged when snapshot released 4. Add simple restore test 5. Use template and constexpr instead of copying codes in PageEntryMap * add test for PageStorage snapshot read * use typed test * Bug fix: Add RefPage to non exist Page * fix bug * Delta merged when view is finished * 1. VersionDeltaSet: add snapshots for snapshot linked-list, so that gc can collect valid PageFiles 2. fix some bugs * do compact on delta -> base * rebase after do compact on delta * fix bugs * avoid duplicted codes * 1. new find API 2. all tests passed * 1. remove legacy API 2. remove duplicated codes * reformat * apply inplace if there are no readers * fix bug of applying edits * fix bug of gc apply * fix bug of merging PageEntryMap * fix bug of invalid ref page * fix bug: 1. listAllLiveFiles 2. double compaction on same version * tmp * fixbug * split PageEntryMapView to single cpp file * minor fix * 1. Avoid visit same version multiple time in PageEntryMapDeltaVersionSet::listAllLiveFiles 2. use template to reduce duplicated code for gcApply * todo mark * fix bug: RefPage -> RefPage record may become invalid record when PageStorage GC * remove unused function * simple fix on unittest * fix bug: PageEntryMapView::isRefId * fix bug: PageEntryMapDeltaBuilder::applyPut * fix * fix bug: PageEntryView::validNormalPageIds filter out tombstone of PageEntry * iterator over PageEntryMap * remove unused tests * Refactor on PageEntryMapDeltaVersionSet * Reduce lock range when PageStorage Snapshot is release * Add metrics for PageStorage MVCC * rename some classes * reformat. * use exception instead * adress comment * adress comment * fix compiler error under gcc * fix compiler error under gcc
JaySon-Huang
added a commit
to JaySon-Huang/tiflash
that referenced
this pull request
Oct 30, 2019
…ingcap#97) * Page storage bug fix (pingcap#87) * add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT * add RefPage && ref counting * 1. move PageCacheMap into isolated file 2. rename PageCacheMap -> PageEntryMap * accept non-exist Page reference/del while being used by WriteBatch or Gc * adjust gc on RefPages * add more test on gc RefPages * add memory usage comparing between vec/hashmap/treemap * fix tests code format * Bug fix: Add RefPage to non exist Page * Add unit test for PageStorage * fix bug * stress test for benchmark * fix bug of PageEntryMap updating RefPage * Keep RefPage entry ref-count if it has been gc and move to another place * MVCC on PageMap, All previse tests passed. TODO: refactor on read_mutex on PageStorage * PageStorage gc: only delete file that are not used by any version * fix bug under multi-threads; remove unused code && format codes * 1. support snapshot read 2. standalone VersionSet/MultiVersionCountable template * 1. Apply merge delta if no read ref 2. Apply generate new delta if has read ref 3. Delta merged when snapshot released 4. Add simple restore test 5. Use template and constexpr instead of copying codes in PageEntryMap * add test for PageStorage snapshot read * use typed test * Bug fix: Add RefPage to non exist Page * fix bug * Delta merged when view is finished * 1. VersionDeltaSet: add snapshots for snapshot linked-list, so that gc can collect valid PageFiles 2. fix some bugs * do compact on delta -> base * rebase after do compact on delta * fix bugs * avoid duplicted codes * 1. new find API 2. all tests passed * 1. remove legacy API 2. remove duplicated codes * reformat * apply inplace if there are no readers * fix bug of applying edits * fix bug of gc apply * fix bug of merging PageEntryMap * fix bug of invalid ref page * fix bug: 1. listAllLiveFiles 2. double compaction on same version * tmp * fixbug * split PageEntryMapView to single cpp file * minor fix * 1. Avoid visit same version multiple time in PageEntryMapDeltaVersionSet::listAllLiveFiles 2. use template to reduce duplicated code for gcApply * todo mark * fix bug: RefPage -> RefPage record may become invalid record when PageStorage GC * remove unused function * simple fix on unittest * fix bug: PageEntryMapView::isRefId * fix bug: PageEntryMapDeltaBuilder::applyPut * fix * fix bug: PageEntryView::validNormalPageIds filter out tombstone of PageEntry * iterator over PageEntryMap * remove unused tests * Refactor on PageEntryMapDeltaVersionSet * Reduce lock range when PageStorage Snapshot is release * Add metrics for PageStorage MVCC * rename some classes * reformat. * use exception instead * adress comment * adress comment * fix compiler error under gcc * fix compiler error under gcc
JaySon-Huang
added a commit
to JaySon-Huang/tiflash
that referenced
this pull request
Nov 1, 2019
…ingcap#97) * Page storage bug fix (pingcap#87) * add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT * add RefPage && ref counting * 1. move PageCacheMap into isolated file 2. rename PageCacheMap -> PageEntryMap * accept non-exist Page reference/del while being used by WriteBatch or Gc * adjust gc on RefPages * add more test on gc RefPages * add memory usage comparing between vec/hashmap/treemap * fix tests code format * Bug fix: Add RefPage to non exist Page * Add unit test for PageStorage * fix bug * stress test for benchmark * fix bug of PageEntryMap updating RefPage * Keep RefPage entry ref-count if it has been gc and move to another place * MVCC on PageMap, All previse tests passed. TODO: refactor on read_mutex on PageStorage * PageStorage gc: only delete file that are not used by any version * fix bug under multi-threads; remove unused code && format codes * 1. support snapshot read 2. standalone VersionSet/MultiVersionCountable template * 1. Apply merge delta if no read ref 2. Apply generate new delta if has read ref 3. Delta merged when snapshot released 4. Add simple restore test 5. Use template and constexpr instead of copying codes in PageEntryMap * add test for PageStorage snapshot read * use typed test * Bug fix: Add RefPage to non exist Page * fix bug * Delta merged when view is finished * 1. VersionDeltaSet: add snapshots for snapshot linked-list, so that gc can collect valid PageFiles 2. fix some bugs * do compact on delta -> base * rebase after do compact on delta * fix bugs * avoid duplicted codes * 1. new find API 2. all tests passed * 1. remove legacy API 2. remove duplicated codes * reformat * apply inplace if there are no readers * fix bug of applying edits * fix bug of gc apply * fix bug of merging PageEntryMap * fix bug of invalid ref page * fix bug: 1. listAllLiveFiles 2. double compaction on same version * tmp * fixbug * split PageEntryMapView to single cpp file * minor fix * 1. Avoid visit same version multiple time in PageEntryMapDeltaVersionSet::listAllLiveFiles 2. use template to reduce duplicated code for gcApply * todo mark * fix bug: RefPage -> RefPage record may become invalid record when PageStorage GC * remove unused function * simple fix on unittest * fix bug: PageEntryMapView::isRefId * fix bug: PageEntryMapDeltaBuilder::applyPut * fix * fix bug: PageEntryView::validNormalPageIds filter out tombstone of PageEntry * iterator over PageEntryMap * remove unused tests * Refactor on PageEntryMapDeltaVersionSet * Reduce lock range when PageStorage Snapshot is release * Add metrics for PageStorage MVCC * rename some classes * reformat. * use exception instead * adress comment * adress comment * fix compiler error under gcc * fix compiler error under gcc
JaySon-Huang
added a commit
to JaySon-Huang/tiflash
that referenced
this pull request
Nov 1, 2019
…ingcap#97) * Page storage bug fix (pingcap#87) * add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT * add RefPage && ref counting * 1. move PageCacheMap into isolated file 2. rename PageCacheMap -> PageEntryMap * accept non-exist Page reference/del while being used by WriteBatch or Gc * adjust gc on RefPages * add more test on gc RefPages * add memory usage comparing between vec/hashmap/treemap * fix tests code format * Bug fix: Add RefPage to non exist Page * Add unit test for PageStorage * fix bug * stress test for benchmark * fix bug of PageEntryMap updating RefPage * Keep RefPage entry ref-count if it has been gc and move to another place * MVCC on PageMap, All previse tests passed. TODO: refactor on read_mutex on PageStorage * PageStorage gc: only delete file that are not used by any version * fix bug under multi-threads; remove unused code && format codes * 1. support snapshot read 2. standalone VersionSet/MultiVersionCountable template * 1. Apply merge delta if no read ref 2. Apply generate new delta if has read ref 3. Delta merged when snapshot released 4. Add simple restore test 5. Use template and constexpr instead of copying codes in PageEntryMap * add test for PageStorage snapshot read * use typed test * Bug fix: Add RefPage to non exist Page * fix bug * Delta merged when view is finished * 1. VersionDeltaSet: add snapshots for snapshot linked-list, so that gc can collect valid PageFiles 2. fix some bugs * do compact on delta -> base * rebase after do compact on delta * fix bugs * avoid duplicted codes * 1. new find API 2. all tests passed * 1. remove legacy API 2. remove duplicated codes * reformat * apply inplace if there are no readers * fix bug of applying edits * fix bug of gc apply * fix bug of merging PageEntryMap * fix bug of invalid ref page * fix bug: 1. listAllLiveFiles 2. double compaction on same version * tmp * fixbug * split PageEntryMapView to single cpp file * minor fix * 1. Avoid visit same version multiple time in PageEntryMapDeltaVersionSet::listAllLiveFiles 2. use template to reduce duplicated code for gcApply * todo mark * fix bug: RefPage -> RefPage record may become invalid record when PageStorage GC * remove unused function * simple fix on unittest * fix bug: PageEntryMapView::isRefId * fix bug: PageEntryMapDeltaBuilder::applyPut * fix * fix bug: PageEntryView::validNormalPageIds filter out tombstone of PageEntry * iterator over PageEntryMap * remove unused tests * Refactor on PageEntryMapDeltaVersionSet * Reduce lock range when PageStorage Snapshot is release * Add metrics for PageStorage MVCC * rename some classes * reformat. * use exception instead * adress comment * adress comment * fix compiler error under gcc * fix compiler error under gcc
JaySon-Huang
added a commit
to JaySon-Huang/tiflash
that referenced
this pull request
Nov 5, 2019
…ingcap#97) * Page storage bug fix (pingcap#87) * add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT * add RefPage && ref counting * 1. move PageCacheMap into isolated file 2. rename PageCacheMap -> PageEntryMap * accept non-exist Page reference/del while being used by WriteBatch or Gc * adjust gc on RefPages * add more test on gc RefPages * add memory usage comparing between vec/hashmap/treemap * fix tests code format * Bug fix: Add RefPage to non exist Page * Add unit test for PageStorage * fix bug * stress test for benchmark * fix bug of PageEntryMap updating RefPage * Keep RefPage entry ref-count if it has been gc and move to another place * MVCC on PageMap, All previse tests passed. TODO: refactor on read_mutex on PageStorage * PageStorage gc: only delete file that are not used by any version * fix bug under multi-threads; remove unused code && format codes * 1. support snapshot read 2. standalone VersionSet/MultiVersionCountable template * 1. Apply merge delta if no read ref 2. Apply generate new delta if has read ref 3. Delta merged when snapshot released 4. Add simple restore test 5. Use template and constexpr instead of copying codes in PageEntryMap * add test for PageStorage snapshot read * use typed test * Bug fix: Add RefPage to non exist Page * fix bug * Delta merged when view is finished * 1. VersionDeltaSet: add snapshots for snapshot linked-list, so that gc can collect valid PageFiles 2. fix some bugs * do compact on delta -> base * rebase after do compact on delta * fix bugs * avoid duplicted codes * 1. new find API 2. all tests passed * 1. remove legacy API 2. remove duplicated codes * reformat * apply inplace if there are no readers * fix bug of applying edits * fix bug of gc apply * fix bug of merging PageEntryMap * fix bug of invalid ref page * fix bug: 1. listAllLiveFiles 2. double compaction on same version * tmp * fixbug * split PageEntryMapView to single cpp file * minor fix * 1. Avoid visit same version multiple time in PageEntryMapDeltaVersionSet::listAllLiveFiles 2. use template to reduce duplicated code for gcApply * todo mark * fix bug: RefPage -> RefPage record may become invalid record when PageStorage GC * remove unused function * simple fix on unittest * fix bug: PageEntryMapView::isRefId * fix bug: PageEntryMapDeltaBuilder::applyPut * fix * fix bug: PageEntryView::validNormalPageIds filter out tombstone of PageEntry * iterator over PageEntryMap * remove unused tests * Refactor on PageEntryMapDeltaVersionSet * Reduce lock range when PageStorage Snapshot is release * Add metrics for PageStorage MVCC * rename some classes * reformat. * use exception instead * adress comment * adress comment * fix compiler error under gcc * fix compiler error under gcc
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Design document is here.
https://docs.google.com/document/d/1kmryprrQTlycbkm-KuSKzA5oYHeRos-YzgOf5l0_OA4/edit#