Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ticdc: add scale out for kafka changefeed #12693

Merged
merged 13 commits into from
Feb 10, 2023
19 changes: 19 additions & 0 deletions ticdc/ticdc-sink-to-kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -233,3 +233,22 @@ partition 分发器用 partition = "xxx" 来指定,支持 default、ts、index
> ```
> {matcher = ['*.*'], dispatcher = "ts", partition = "table"},
> ```

## 横向扩展大单表的负载到多个 TiCDC 节点

该功能可以解决以下两个问题:
shichun-0415 marked this conversation as resolved.
Show resolved Hide resolved
overvenus marked this conversation as resolved.
Show resolved Hide resolved

- 单个 TiCDC 节点不能及时同步大单表的问题
- TiCDC 节点之间资源(CPU、内存等)消耗不均匀的问题。
overvenus marked this conversation as resolved.
Show resolved Hide resolved
> **注意:**
overvenus marked this conversation as resolved.
Show resolved Hide resolved
>
> TiCDC v6.6.0 仅支持在 Kafka 同步任务上开启大单表的水平扩展功能。
overvenus marked this conversation as resolved.
Show resolved Hide resolved

overvenus marked this conversation as resolved.
Show resolved Hide resolved
该功能通过将大单表按 Region 个数切分成多个数据范围,将这些数据范围分布到多个 TiCDC 节点上,使得多个 TiCDC 节点可以同时同步大单表。
Copy link
Contributor

@3AceShowHand 3AceShowHand Jan 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使得多个 TiCDC 节点可以同时同步大单表。-> 使得多个 TiCDC 节点,可以同时单张表的数据,在处理大单表场景时,可以提升大单表的同步吞吐量。

overvenus marked this conversation as resolved.
Show resolved Hide resolved

配置样例如下所示:

```toml
[scheduler]
region-per-span = 50000
```