cdc changefeed stucks

[fubinzh]

What did you do?

1. deploy tidb cluster with 3 CDC
2. create 2 kafka changefeed, one canal json, the other open protocal
3. run sysbench prepare
   sysbench --db-driver=mysql --mysql-host=`nslookup upstream-tidb.cdc-testbed-tps-1530295-1-204 | awk -F: '{print $2}' | awk 'NR==5' | sed s/[[:space:]]//g` --mysql-port=4000 --mysql-user=root --mysql-db=workload --tables=30 --table-size=1000000 --create_secondary=off --debug=true --threads=20 --mysql-ignore-errors=2013,1213,1105,1205,8022,8027,8028,9004,9007,1062 oltp_write_only prepare
4. scale out tikv to 6
5. scale in ticdc to 1
6. run sysbench, and at the same time, scale tikv to 3, scale cdc to 6

ysbench --db-driver=mysql --mysql-host=`nslookup upstream-tidb.cdc-testbed-tps-1530295-1-204 | awk -F: '{print $2}' | awk 'NR==5' | sed s/[[:space:]]//g`  --mysql-port=4000 --mysql-user=root --mysql-db=workload --tables=30 --table-size=1000000 --create_secondary=off --time=7200 --debug=true --threads=20 --mysql-ignore-errors=2013,1213,1105,1205,8022,8027,8028,9004,9007,1062 oltp_write_only run

8. wait cdc changefeed advance

What did you expect to see?

cdc changefeed should advance

What did you see instead?

• canal json changefeed advances. but open protocal changefeed stucks.

Versions of the cluster

CDC version:
[2023/02/04 01:53:09.767 +00:00] [INFO] [version.go:47] ["Welcome to Change Data Capture (CDC)"] [release-version=v6.6.0-alpha] [git-hash=84a76a88348b2bbf760a81e42ef2f5a3b0182dbc] [git-branch=heads/refs/tags/v6.6.0-alpha] [utc-build-time="2023-02-03 09:46:49"] [go-version="go version go1.19.5 linux/amd64"] [failpoint-build=false]

[fubinzh]

clinic: https://clinic.pingcap.com.cn/portal/#/orgs/31/clusters/7196112540470299131?from=1675475476&to=1675501114

[fubinzh]

/severity Critical

[nongfushanquan]

/assign @3AceShowHand

[fubinzh]

/found automation