Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ticdc: update scale out config #13578

Merged
merged 7 commits into from
May 17, 2023
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion ticdc/ticdc-changefeed-config.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,8 +94,11 @@ ignore-insert-value-expr = "price > 1000 and origin = 'no where'" # 忽略包含
# 注意:该功能只在 Kafka changefeed 上生效,暂不支持 MySQL changefeed。
# 默认为 "false"。设置为 "true" 以打开该功能。
enable-table-across-nodes = false
# 打开该功能后,该功能只对 Region 个数大于 `region-threshold` 值的表生效。
# 打开该功能后,该功能会对 Region 个数大于 `region-threshold` 值的表生效。
region-threshold = 100000
# 打开该功能后,该功能会对每分钟修改行数大于 `write-key-threshold` 值的表生效。
# 注意:该参数默认值为 0,代表该功能默认不会按修改行数来划分表。
overvenus marked this conversation as resolved.
Show resolved Hide resolved
write-key-threshold = 0

[sink]
# 对于 MQ 类的 Sink,可以通过 dispatchers 配置 event 分发器
Expand Down
5 changes: 4 additions & 1 deletion ticdc/ticdc-sink-to-kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,7 @@ partition 分发器用 partition = "xxx" 来指定,支持 default、ts、index

## 横向扩展大单表的负载到多个 TiCDC 节点

该功能通过将大单表按 Region 个数切分成多个数据范围,将这些数据范围分布到多个 TiCDC 节点上,使得多个 TiCDC 节点可以同时同步大单表。该功能可以解决以下两个问题:
该功能通过将大单表按数据量和每分钟修改行数切分成多个数据范围,使得数据范围之间的数据量和每分钟修改行数基本相同。该功能将这些数据范围分布到多个 TiCDC 节点上,使得多个 TiCDC 节点可以同时同步大单表。该功能可以解决以下两个问题:
overvenus marked this conversation as resolved.
Show resolved Hide resolved

- 单个 TiCDC 节点不能及时同步大单表。
- TiCDC 节点之间资源(CPU、内存等)消耗不均匀。
Expand All @@ -254,6 +254,9 @@ partition 分发器用 partition = "xxx" 来指定,支持 default、ts、index
enable-table-across-nodes = true
overvenus marked this conversation as resolved.
Show resolved Hide resolved
# 打开该功能后,该功能只对 Region 个数大于 `region-threshold` 值的表生效。
region-threshold = 100000
# 打开该功能后,该功能会对每分钟修改行数大于 `write-key-threshold` 值的表生效。
# 注意:该参数默认值为 0,代表该功能默认不会按修改行数来划分表。
write-key-threshold = 0
overvenus marked this conversation as resolved.
Show resolved Hide resolved
overvenus marked this conversation as resolved.
Show resolved Hide resolved
```

一个表包含的 Region 个数可用如下 SQL 查询:
Expand Down