Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ticdc: add scale out for kafka changefeed #12693

Merged
merged 13 commits into from
Feb 10, 2023
Merged

ticdc: add scale out for kafka changefeed #12693

merged 13 commits into from
Feb 10, 2023

Conversation

overvenus
Copy link
Member

@overvenus overvenus commented Jan 17, 2023 •
edited
Loading

First-time contributors' checklist

What is changed, added or deleted? (Required)

Add doc about scale out feature for kafka changefeed.

Which TiDB version(s) do your changes apply to? (Required)

Tips for choosing the affected version(s):

By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.

For details, see tips for choosing the affected versions (in Chinese).

  • master (the latest development version)
  • v6.6 (TiDB 6.6 versions)
  • v6.5 (TiDB 6.5 versions)
  • v6.4 (TiDB 6.4 versions)
  • v6.1 (TiDB 6.1 versions)
  • v5.4 (TiDB 5.4 versions)
  • v5.3 (TiDB 5.3 versions)
  • v5.2 (TiDB 5.2 versions)
  • v5.1 (TiDB 5.1 versions)
  • v5.0 (TiDB 5.0 versions)

What is the related PR or file link(s)?

Do your changes match any of the following descriptions?

  • Delete files
  • Change aliases
  • Need modification after applied to another branch
  • Might cause conflicts after applied to another branch

@overvenus
ticdc: add scale out for kafka changefeed
492b8d7 
Signed-off-by: Neil Shen <overvenus@gmail.com>
@overvenus overvenus added the area/ticdc Indicates that the Issue or PR belongs to the area of TiCDC. label Jan 17, 2023
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jan 17, 2023 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • Oreoxmt

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot requested a review from Oreoxmt January 17, 2023 12:59
@ti-chi-bot ti-chi-bot added missing-translation-status This PR does not have translation status info. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jan 17, 2023
@overvenus
break lines
19c3a56 
Signed-off-by: Neil Shen <overvenus@gmail.com>
@Oreoxmt Oreoxmt added the translation/doing This PR’s assignee is translating this PR. label Jan 17, 2023
@ti-chi-bot ti-chi-bot removed the missing-translation-status This PR does not have translation status info. label Jan 17, 2023
@Oreoxmt Oreoxmt added the v6.6 label Jan 17, 2023
@overvenus
increase head
cf745f8 
Signed-off-by: Neil Shen <overvenus@gmail.com>
@overvenus
typo
1f4dd48 
Signed-off-by: Neil Shen <overvenus@gmail.com>
@Oreoxmt Oreoxmt self-assigned this Jan 17, 2023
@Oreoxmt
Copy link
Collaborator

Oreoxmt commented Jan 17, 2023

@overvenus Could you please invite a tech reviewer? Thanks!

@overvenus overvenus mentioned this pull request Jan 19, 2023
Closed
22 tasks
@Frank945946 Frank945946 mentioned this pull request Jan 20, 2023
Merged
14 tasks
Oreoxmt
Oreoxmt reviewed Jan 20, 2023
View reviewed changes
Copy link
Collaborator

@Oreoxmt Oreoxmt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM

ticdc/ticdc-sink-to-kafka.md Outdated Show resolved Hide resolved
ticdc/ticdc-sink-to-kafka.md Outdated Show resolved Hide resolved
ticdc/ticdc-sink-to-kafka.md Outdated Show resolved Hide resolved
@Oreoxmt Oreoxmt self-requested a review January 20, 2023 04:04
@Oreoxmt
Copy link
Collaborator

Oreoxmt commented Jan 28, 2023

@3AceShowHand PTAL

3AceShowHand
3AceShowHand reviewed Jan 28, 2023
View reviewed changes
ticdc/ticdc-sink-to-kafka.md Outdated
@@ -233,3 +233,20 @@ partition 分发器用 partition = "xxx" 来指定，支持 default、ts、index
> ```
> {matcher = ['*.*'], dispatcher = "ts", partition = "table"},
> ```

## 将大单表的同步负载水平扩展到多个 TiCDC 节点上
Copy link
Contributor

@3AceShowHand 3AceShowHand Jan 28, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

横向扩展大单表的负载到多个 TiCDC 节点

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

水平 -> 横向

3AceShowHand
3AceShowHand reviewed Jan 28, 2023
View reviewed changes
ticdc/ticdc-sink-to-kafka.md Outdated

## 将大单表的同步负载水平扩展到多个 TiCDC 节点上

该功能可以解决单个 TiCDC 节点不能及时同步大单表的问题和 TiCDC 节点之间资源（CPU，内存等）消耗出现倾斜的问题。
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

资源消耗倾斜 -> 不均匀

3AceShowHand
3AceShowHand reviewed Jan 28, 2023
View reviewed changes
ticdc/ticdc-sink-to-kafka.md Outdated

## 将大单表的同步负载水平扩展到多个 TiCDC 节点上

该功能可以解决单个 TiCDC 节点不能及时同步大单表的问题和 TiCDC 节点之间资源（CPU，内存等）消耗出现倾斜的问题。
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这句话包含了两个场景，建议两个场景用子弹列表分开描述。

@overvenus @Oreoxmt
Apply suggestions from code review
701403b 
Co-authored-by: Aolin <aolinz@outlook.com>
3AceShowHand
3AceShowHand reviewed Jan 28, 2023
View reviewed changes
ticdc/ticdc-sink-to-kafka.md Outdated
- 单个 TiCDC 节点不能及时同步大单表的问题
- TiCDC 节点之间资源（CPU、内存等）消耗不均匀的问题。

该功能通过将大单表按 Region 个数切分成多个数据范围，将这些数据范围分布到多个 TiCDC 节点上，使得多个 TiCDC 节点可以同时同步大单表。
Copy link
Contributor

@3AceShowHand 3AceShowHand Jan 28, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使得多个 TiCDC 节点可以同时同步大单表。-> 使得多个 TiCDC 节点，可以同时单张表的数据，在处理大单表场景时，可以提升大单表的同步吞吐量。

Oreoxmt
Oreoxmt reviewed Jan 28, 2023
View reviewed changes
ticdc/ticdc-sink-to-kafka.md Show resolved Hide resolved
ticdc/ticdc-sink-to-kafka.md Outdated Show resolved Hide resolved
@Oreoxmt
Copy link
Collaborator

Oreoxmt commented Jan 28, 2023

/status LGT1

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jan 28, 2023
@overvenus @Oreoxmt
Apply suggestions from code review
a63df6a 
Co-authored-by: Aolin <aolinz@outlook.com>
Oreoxmt
Oreoxmt approved these changes Jan 28, 2023
View reviewed changes
ticdc/ticdc-sink-to-kafka.md Show resolved Hide resolved
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jan 28, 2023
@Oreoxmt Oreoxmt requested a review from shichun-0415 January 28, 2023 09:33
@shichun-0415 shichun-0415 added the type/compatibility-or-feature-change This PR involves compatibility changes or feature behavior changes. label Jan 28, 2023
shichun-0415
shichun-0415 reviewed Jan 28, 2023
View reviewed changes
ticdc/ticdc-sink-to-kafka.md Outdated Show resolved Hide resolved
ticdc/ticdc-sink-to-kafka.md Outdated Show resolved Hide resolved
@overvenus @shichun-0415 @Oreoxmt
Apply suggestions from code review
6c48ebf 
Co-authored-by: shichun-0415 <89768198+shichun-0415@users.noreply.github.com>
Co-authored-by: Aolin <aolinz@outlook.com>
overvenus
overvenus commented Jan 29, 2023
View reviewed changes
ticdc/ticdc-sink-to-kafka.md Outdated Show resolved Hide resolved
ticdc/ticdc-sink-to-kafka.md Outdated Show resolved Hide resolved
@Oreoxmt Oreoxmt requested a review from shichun-0415 February 1, 2023 06:38
Oreoxmt
Oreoxmt approved these changes Feb 1, 2023
View reviewed changes
Copy link
Collaborator

@Oreoxmt Oreoxmt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM

ticdc/ticdc-sink-to-kafka.md Outdated Show resolved Hide resolved
@Oreoxmt Oreoxmt mentioned this pull request Feb 1, 2023
Merged
14 tasks
@Oreoxmt
Copy link
Collaborator

Oreoxmt commented Feb 1, 2023

@overvenus 是否需要在 TiCDC Changefeed 相关的配置项说明文档中添加 region-per-span 的说明？ https://docs.pingcap.com/zh/tidb/stable/ticdc-changefeed-config

@Oreoxmt Oreoxmt added translation/done This PR has been translated from English into Chinese and updated to pingcap/docs-cn in a PR. and removed translation/doing This PR’s assignee is translating this PR. labels Feb 1, 2023
overvenus and others added 2 commits February 2, 2023 19:52
@overvenus @Oreoxmt
Update ticdc/ticdc-sink-to-kafka.md
28fc122 
Co-authored-by: Aolin <aolinz@outlook.com>
@overvenus
address comments
1d6b43d 
Signed-off-by: Neil Shen <overvenus@gmail.com>
shichun-0415
shichun-0415 approved these changes Feb 8, 2023
View reviewed changes
Copy link
Contributor

@shichun-0415 shichun-0415 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Oreoxmt
Copy link
Collaborator

Oreoxmt commented Feb 10, 2023

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 1d6b43d

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Feb 10, 2023
@ti-chi-bot ti-chi-bot merged commit 4b3c34f into pingcap:master Feb 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shichun-0415 shichun-0415 shichun-0415 approved these changes

@Oreoxmt Oreoxmt Oreoxmt approved these changes

@3AceShowHand 3AceShowHand 3AceShowHand approved these changes

Assignees

@Oreoxmt Oreoxmt

Labels
area/ticdc Indicates that the Issue or PR belongs to the area of TiCDC. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. translation/done This PR has been translated from English into Chinese and updated to pingcap/docs-cn in a PR. type/compatibility-or-feature-change This PR involves compatibility changes or feature behavior changes. v6.6
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@overvenus @ti-chi-bot @Oreoxmt @3AceShowHand @shichun-0415