Refactor WAL log writer && IO limiter #4107

jiaqizho · 2022-02-23T08:33:02Z

What problem does this PR solve?

Issue Number: ref #3594

Problem Summary:

What is changed and how it works?

PageStorage will pass IOLimiter when call write/read
for now, WALStore only supports passing IOLimiter in the constructor.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

ti-chi-bot · 2022-02-23T08:33:03Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

JaySon-Huang

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: jiaqizho <zhoujiaqi@pingcap.com>

dbms/src/Storages/Page/V3/PageDirectory.h

Signed-off-by: jiaqizho <zhoujiaqi@pingcap.com>

JaySon-Huang

LGTM with minor comment

dbms/src/Storages/Page/V3/LogFile/LogWriter.h

jiaqizho · 2022-03-01T06:08:18Z

/merge

ti-chi-bot · 2022-03-01T06:08:19Z

@jiaqizho: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-03-01T06:35:54Z

@jiaqizho: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

dbms/src/Storages/Page/V3/WALStore.cpp

JaySon-Huang · 2022-03-01T08:50:47Z

/merge cancel

jiaqizho · 2022-03-01T08:56:21Z

/run-all-tests

sre-bot · 2022-03-01T09:07:01Z

Coverage for changed files

Filename                                      Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
LogFile/LogWriter.cpp                              50                 1    98.00%           8                 1    87.50%         109                 8    92.66%          32                 0   100.00%
LogFile/LogWriter.h                                 1                 0   100.00%           1                 0   100.00%           3                 0   100.00%           0                 0         -
PageDirectory.cpp                                 183                41    77.60%          23                 4    82.61%         431                98    77.26%         138                36    73.91%
PageDirectory.h                                    16                 4    75.00%          14                 4    71.43%          52                15    71.15%           2                 1    50.00%
PageStorageImpl.cpp                                62                26    58.06%          18                10    44.44%         146                73    50.00%          32                21    34.38%
WAL/WALReader.cpp                                  44                 6    86.36%           7                 0   100.00%         106                 9    91.51%          30                 6    80.00%
WAL/WALReader.h                                     9                 2    77.78%           4                 1    75.00%          17                 6    64.71%           4                 1    75.00%
WALStore.cpp                                       70                 4    94.29%           7                 0   100.00%         116                 0   100.00%          34                 7    79.41%
tests/gtest_collapsing_page_directory.cpp         361                77    78.67%           4                 0   100.00%          95                 0   100.00%         110                52    52.73%
tests/gtest_wal_log.cpp                          2517               370    85.30%          49                 0   100.00%         532                 1    99.81%         768               351    54.30%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                            3313               531    83.97%         135                20    85.19%        1607               210    86.93%        1150               475    58.70%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16691      9466             43.29%    186700  95614        48.79%

full coverage report (for internal network access only)

jiaqizho · 2022-03-01T09:42:23Z

/merge

ti-chi-bot · 2022-03-01T09:42:23Z

@jiaqizho: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-03-01T09:42:29Z

This pull request has been accepted and is ready to merge.

Commit hash: 5379c6e

sre-bot · 2022-03-01T09:54:29Z

Coverage for changed files

Filename                                      Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
LogFile/LogWriter.cpp                              50                 1    98.00%           8                 1    87.50%         109                 8    92.66%          32                 0   100.00%
LogFile/LogWriter.h                                 1                 0   100.00%           1                 0   100.00%           3                 0   100.00%           0                 0         -
PageDirectory.cpp                                 183                41    77.60%          23                 4    82.61%         431                98    77.26%         138                36    73.91%
PageDirectory.h                                    16                 4    75.00%          14                 4    71.43%          52                15    71.15%           2                 1    50.00%
PageStorageImpl.cpp                                62                26    58.06%          18                10    44.44%         146                73    50.00%          32                21    34.38%
WAL/WALReader.cpp                                  44                 6    86.36%           7                 0   100.00%         106                 9    91.51%          30                 6    80.00%
WAL/WALReader.h                                     9                 2    77.78%           4                 1    75.00%          17                 6    64.71%           4                 1    75.00%
WALStore.cpp                                       70                 4    94.29%           7                 0   100.00%         116                 0   100.00%          34                 7    79.41%
tests/gtest_collapsing_page_directory.cpp         361                77    78.67%           4                 0   100.00%          95                 0   100.00%         110                52    52.73%
tests/gtest_wal_log.cpp                          2517               370    85.30%          49                 0   100.00%         532                 1    99.81%         768               351    54.30%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                            3313               531    83.97%         135                20    85.19%        1607               210    86.93%        1150               475    58.70%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16691      9466             43.29%    186700  95626        48.78%

full coverage report (for internal network access only)

jiaqizho · 2022-03-02T02:14:11Z

/merge

ti-chi-bot · 2022-03-02T02:14:12Z

@jiaqizho: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

sre-bot · 2022-03-02T02:24:07Z

Coverage for changed files

Filename                                      Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
LogFile/LogWriter.cpp                              50                 1    98.00%           8                 1    87.50%         109                 8    92.66%          32                 0   100.00%
LogFile/LogWriter.h                                 1                 0   100.00%           1                 0   100.00%           3                 0   100.00%           0                 0         -
PageDirectory.cpp                                 183                41    77.60%          23                 4    82.61%         431                98    77.26%         138                36    73.91%
PageDirectory.h                                    16                 4    75.00%          14                 4    71.43%          52                15    71.15%           2                 1    50.00%
PageStorageImpl.cpp                                62                26    58.06%          18                10    44.44%         146                73    50.00%          32                21    34.38%
WAL/WALReader.cpp                                  44                 6    86.36%           7                 0   100.00%         106                 9    91.51%          30                 6    80.00%
WAL/WALReader.h                                     9                 2    77.78%           4                 1    75.00%          17                 6    64.71%           4                 1    75.00%
WALStore.cpp                                       70                 4    94.29%           7                 0   100.00%         116                 0   100.00%          34                 7    79.41%
tests/gtest_collapsing_page_directory.cpp         361                77    78.67%           4                 0   100.00%          95                 0   100.00%         110                52    52.73%
tests/gtest_wal_log.cpp                          2517               370    85.30%          49                 0   100.00%         532                 1    99.81%         768               351    54.30%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                            3313               531    83.97%         135                20    85.19%        1607               210    86.93%        1150               475    58.70%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16691      9467             43.28%    186700  95632        48.78%

full coverage report (for internal network access only)

[WIP]Change WAL log writer to writable file

e5affb1

update

1566ca6

Signed-off-by: jiaqizho <zhoujiaqi@pingcap.com>

JaySon-Huang reviewed Feb 24, 2022

View reviewed changes

dbms/src/Storages/Page/V3/PageDirectory.h Show resolved Hide resolved

fix tests

ec48fef

ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Feb 24, 2022

fix tests

930b1af

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Feb 24, 2022

jiaqizho changed the title ~~[WIP]Change WAL log writer to writable file~~ Refactor WAL log writer && IO limiter Feb 24, 2022

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 24, 2022

jiaqizho added 2 commits February 24, 2022 16:19

update

5ef1e10

Signed-off-by: jiaqizho <zhoujiaqi@pingcap.com>

update

95ebd67

Signed-off-by: jiaqizho <zhoujiaqi@pingcap.com>

ti-chi-bot removed the do-not-merge/needs-linked-issue label Feb 24, 2022

JaySon-Huang approved these changes Mar 1, 2022

View reviewed changes

dbms/src/Storages/Page/V3/LogFile/LogWriter.h Outdated Show resolved Hide resolved

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 1, 2022

update

9d4cdf9

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 1, 2022

JaySon-Huang reviewed Mar 1, 2022

View reviewed changes

dbms/src/Storages/Page/V3/WALStore.cpp Outdated Show resolved Hide resolved

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Mar 1, 2022

fix

727705d

Merge branch 'master' into wal-replace-writablefile

5379c6e

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 1, 2022

ti-chi-bot merged commit a89b3bb into pingcap:master Mar 2, 2022

JaySon-Huang deleted the wal-replace-writablefile branch March 2, 2022 02:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor WAL log writer && IO limiter #4107

Refactor WAL log writer && IO limiter #4107

jiaqizho commented Feb 23, 2022 •

edited by JaySon-Huang

Loading

ti-chi-bot commented Feb 23, 2022 •

edited

Loading

JaySon-Huang left a comment

jiaqizho commented Mar 1, 2022

ti-chi-bot commented Mar 1, 2022

ti-chi-bot commented Mar 1, 2022

JaySon-Huang commented Mar 1, 2022

jiaqizho commented Mar 1, 2022

sre-bot commented Mar 1, 2022

jiaqizho commented Mar 1, 2022

ti-chi-bot commented Mar 1, 2022

ti-chi-bot commented Mar 1, 2022

sre-bot commented Mar 1, 2022

jiaqizho commented Mar 2, 2022

ti-chi-bot commented Mar 2, 2022

sre-bot commented Mar 2, 2022

Refactor WAL log writer && IO limiter #4107

Refactor WAL log writer && IO limiter #4107

Conversation

jiaqizho commented Feb 23, 2022 • edited by JaySon-Huang Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Feb 23, 2022 • edited Loading

JaySon-Huang left a comment

Choose a reason for hiding this comment

jiaqizho commented Mar 1, 2022

ti-chi-bot commented Mar 1, 2022

ti-chi-bot commented Mar 1, 2022

JaySon-Huang commented Mar 1, 2022

jiaqizho commented Mar 1, 2022

sre-bot commented Mar 1, 2022

Coverage for changed files

Coverage summary

jiaqizho commented Mar 1, 2022

ti-chi-bot commented Mar 1, 2022

ti-chi-bot commented Mar 1, 2022

sre-bot commented Mar 1, 2022

Coverage for changed files

Coverage summary

jiaqizho commented Mar 2, 2022

ti-chi-bot commented Mar 2, 2022

sre-bot commented Mar 2, 2022

Coverage for changed files

Coverage summary

jiaqizho commented Feb 23, 2022 •

edited by JaySon-Huang

Loading

ti-chi-bot commented Feb 23, 2022 •

edited

Loading