[ticdc 5.3.0] tikv_scale_plan_tidb case fail: 1. cdc oom, 2. data inconsistency, 3. changefeed stuck

[Tammyxia]

What did you do?

// create 1 changefeed, 3 cdc, 3 tikv
// run tpcc 100 warehouse prepare
// run tpcc run, meanwhile, scale cluster tikv from 3 -> 7, after 15m, scale in to 3
// create table "finishmark"
// wait table "finishmark" sync to downstream
//check data consistency

What did you expect to see?

pass

What did you see instead?

• 1. cdc oom
• 2. The two table has data inconsistency: warehouse，order_line
• 3. changefeed stuck at almost the time create table "finishmark"

testbed saved for 24 hours.

• upstream grafana: http://172.16.4.180:30687/

• testbed: cdc-testbed--tps-450050-1-772

• conprof: http://172.16.4.180:30857/?query=%7Bnamespace%3D%22cdc-testbed--tps-450050-1-772%22%2C+job%3D%22ticdc%22%2C+profile_path%3D%22%2Fdebug%2Fpprof%2Fheap%22%7D&from=1637135181536&to=1637135672266&now=false

• test case log: http://172.16.4.180:31714/workflows/testground/plan-exec-450050?tab=workflow&nodeId=plan-exec-450050-3513432500&sidePanel=logs

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

TiKV 
Release Version:   5.3.0
Edition:           Community
Git Commit Hash:   d514230a40974393297050645c223bcf1db9aedc
Git Commit Branch: heads/refs/tags/v5.3.0
UTC Build Time:    2021-11-16 12:18:25
Rust Version:      rustc 1.56.0-nightly (2faabf579 2021-07-27)
Enable Features:   jemalloc mem-profiling portable sse protobuf-codec test-engines-rocksdb cloud-aws cloud-gcp
Profile:           dist_release

TiCDC version (execute cdc version):

/cdc version
Release Version: v5.3.0
Git Commit Hash: f847b331572379527bf37a7f19be20448a74b2c2
Git Branch: heads/refs/tags/v5.3.0
UTC Build Time: 2021-11-16 11:54:34
Go Version: go version go1.16.4 linux/amd64
Failpoint Build: false

[Tammyxia]

upstream tidb:
mysql> select count(1) from workload.order_line;
+----------+
| count(1) |
+----------+
| 41568034 |
+----------+

downstream tidb:
mysql> select count(1) from workload.order_line;
+----------+
| count(1) |
+----------+
| 25413084 |
+----------+

almost 2:1

warehouse row count is the same, only columne w_ytd totally different.

[overvenus]

CDC OOM is similar to https://github.com/pingcap/ticdc/pull/3439, has merged to master and will be fixed in next releases.

[overvenus]

#3540 has mitigated this issue, change to severity/major.

[overvenus]

This issue is resolved by https://github.com/pingcap/ticdc/pull/3540 and https://github.com/pingcap/ticdc/pull/3436