domain: support plan replayer dump workers running in concurrency #39347

Yisaer · 2022-11-23T11:40:08Z

Signed-off-by: yisaer disxiaofei@163.com

What problem does this PR solve?

Issue Number: ref #38779

Problem Summary:

What is changed and how it works?

support plan replayer dump workers running in concurrency

Check List

Tests

Unit test
Manual test (add detailed scripts or steps below)

https://pingcap.feishu.cn/docx/Ak9WdC71No5PybxifvocUK59nRX?from=from_lark_index_search

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: yisaer <disxiaofei@163.com>

ti-chi-bot · 2022-11-23T11:40:09Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

qw4990
xuyifangreeneyes

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: yisaer <disxiaofei@163.com>

sre-bot · 2022-11-28T07:05:20Z

All committers have signed the CLA.

qw4990 · 2022-11-28T07:39:37Z

config/config.go

+	StatsLoadConcurrency              uint   `toml:"stats-load-concurrency" json:"stats-load-concurrency"`
+	StatsLoadQueueSize                uint   `toml:"stats-load-queue-size" json:"stats-load-queue-size"`
+	AnalyzePartitionConcurrencyQuota  uint   `toml:"analyze-partition-concurrency-quota" json:"analyze-partition-concurrency-quota"`
+	PlanReplayerDumpWorkerConcurrency uint   `toml:"plan-replayer-dump-worker-concurrency" json:"plan-replayer-dump-worker-concurrency"`


How about using a SQL variable instead of a config item? SQL variables are much easier to use.

we can't adjust worker count in runtime thus we use config.

xuyifangreeneyes · 2022-12-01T07:55:29Z

domain/domain.go

 		}
+		<-do.exit
+		do.planReplayerHandle.planReplayerTaskDumpHandle.Close()


Do we need to also put Close in defer?

qw4990 · 2022-12-06T10:09:27Z

/merge

ti-chi-bot · 2022-12-06T10:09:32Z

This pull request has been accepted and is ready to merge.

Commit hash: 12e838a158f17943186a76a74fff1987f7d5cc49

Yisaer · 2022-12-06T15:52:14Z

/merge

Yisaer · 2022-12-07T04:17:54Z

/merge

ti-chi-bot · 2022-12-07T04:17:58Z

This pull request has been accepted and is ready to merge.

Commit hash: dd20da9

Yisaer · 2022-12-07T04:37:49Z

/run-check_dev_2

sre-bot · 2022-12-07T06:04:30Z

TiDB MergeCI notify

🔴 Bad News! New failing [1] after this pr merged.
These new failed integration tests seem to be caused by the current PR, please try to fix these new failed integration tests, thanks!

CI Name	Result	Duration	Compare with Parent commit
idc-jenkins-ci-tidb/common-test	🟥 failed 1, success 10, total 11	11 min	New failing
idc-jenkins-ci/integration-cdc-test	🔴 failed 1, success 39, total 40	21 min	Existing failure
idc-jenkins-ci-tidb/integration-common-test	✅ all 17 tests passed	20 min	Fixed
idc-jenkins-ci-tidb/sqllogic-test-1	🟢 all 26 tests passed	6 min 56 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2	🟢 all 28 tests passed	6 min 56 sec	Existing passed
idc-jenkins-ci-tidb/tics-test	🟢 all 1 tests passed	5 min 56 sec	Existing passed
idc-jenkins-ci-tidb/integration-ddl-test	🟢 all 6 tests passed	5 min 35 sec	Existing passed
idc-jenkins-ci-tidb/mybatis-test	🟢 all 1 tests passed	4 min 7 sec	Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test	🟢 all 1 tests passed	3 min 18 sec	Existing passed
idc-jenkins-ci-tidb/plugin-test	🟢 build success, plugin test success	4min	Existing passed

support

42ad0d6

Signed-off-by: yisaer <disxiaofei@163.com>

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 23, 2022

Yisaer added 4 commits November 24, 2022 11:01

support

3946c7e

Signed-off-by: yisaer <disxiaofei@163.com>

support

f7efd5e

Signed-off-by: yisaer <disxiaofei@163.com>

support

227345b

Signed-off-by: yisaer <disxiaofei@163.com>

support

d002af8

Signed-off-by: yisaer <disxiaofei@163.com>

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 24, 2022

add log

94b8373

Signed-off-by: yisaer <disxiaofei@163.com>

Yisaer marked this pull request as ready for review November 24, 2022 06:44

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 24, 2022

Yisaer requested review from qw4990 and xuyifangreeneyes November 28, 2022 03:37

Yisaer added 2 commits November 28, 2022 14:17

fix

630fab2

Signed-off-by: yisaer <disxiaofei@163.com>

fix

3a908f9

Signed-off-by: yisaer <disxiaofei@163.com>

qw4990 reviewed Nov 28, 2022

View reviewed changes

Yisaer requested a review from qw4990 November 28, 2022 09:23

xuyifangreeneyes reviewed Dec 1, 2022

View reviewed changes

ti-chi-bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Dec 2, 2022

Yisaer force-pushed the support_multi_dump_worker branch from 7940d36 to 63ad248 Compare December 5, 2022 04:59

xuyifangreeneyes approved these changes Dec 6, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 6, 2022

qw4990 approved these changes Dec 6, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 6, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 6, 2022

fix stats

164f958

Yisaer force-pushed the support_multi_dump_worker branch from 15df29e to 164f958 Compare December 7, 2022 04:15

ti-chi-bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed status/can-merge Indicates a PR has been approved by a committer. labels Dec 7, 2022

fix

dd20da9

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 7, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 7, 2022

Merge branch 'master' into support_multi_dump_worker

c42bcc8

Merge branch 'master' into support_multi_dump_worker

d54ede0

ti-chi-bot merged commit a0a16d7 into pingcap:master Dec 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

domain: support plan replayer dump workers running in concurrency #39347

domain: support plan replayer dump workers running in concurrency #39347

Yisaer commented Nov 23, 2022 •

edited

Loading

ti-chi-bot commented Nov 23, 2022 •

edited

Loading

sre-bot commented Nov 28, 2022 •

edited

Loading

qw4990 Nov 28, 2022

Yisaer Nov 28, 2022

xuyifangreeneyes Dec 1, 2022

Yisaer Dec 5, 2022

qw4990 commented Dec 6, 2022

ti-chi-bot commented Dec 6, 2022

Yisaer commented Dec 6, 2022

Yisaer commented Dec 7, 2022

ti-chi-bot commented Dec 7, 2022

Yisaer commented Dec 7, 2022

sre-bot commented Dec 7, 2022

domain: support plan replayer dump workers running in concurrency #39347

domain: support plan replayer dump workers running in concurrency #39347

Conversation

Yisaer commented Nov 23, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Nov 23, 2022 • edited Loading

sre-bot commented Nov 28, 2022 • edited Loading

qw4990 Nov 28, 2022

Choose a reason for hiding this comment

Yisaer Nov 28, 2022

Choose a reason for hiding this comment

xuyifangreeneyes Dec 1, 2022

Choose a reason for hiding this comment

Yisaer Dec 5, 2022

Choose a reason for hiding this comment

qw4990 commented Dec 6, 2022

ti-chi-bot commented Dec 6, 2022

Yisaer commented Dec 6, 2022

Yisaer commented Dec 7, 2022

ti-chi-bot commented Dec 7, 2022

Yisaer commented Dec 7, 2022

sre-bot commented Dec 7, 2022

TiDB MergeCI notify

Yisaer commented Nov 23, 2022 •

edited

Loading

ti-chi-bot commented Nov 23, 2022 •

edited

Loading

sre-bot commented Nov 28, 2022 •

edited

Loading