Storage sink (csv protocol) data inconsistency after CDC scale and owner switch

[fubinzh]

What did you do?

1. Create storage sink for csv protocol

bash-5.1# cat /tmp/changefeed.toml
[sink]
protocol='csv'
[sink.csv]
include-commit-ts=true

# cdc  cli  changefeed  create "--server=127.0.0.1:8301" "--sink-uri=s3://tmp/ticdc-cloud-storage-test/storage-cdc-scale-sync-cgv1vrqi0jpfd0985tr0?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http://minio.pingcap.net:9001&force-path-style=true&enable-tidb-extension=true" "--changefeed-id=storage-cdc-scale-sync-cgv1vrqi0jpfd0985tr0" "--config=/tmp/changefeed.toml"

3. Run storage consumer to consume the data to downstream
4. Run sysbench prepare

sysbench --db-driver=mysql --mysql-host=`nslookup upstream-tidb.cdc-testbed-tps-1687429-1-63 | awk -F: '{print $2}' | awk 'NR==5' | sed s/[[:space:]]//g`  --mysql-port=4000 --mysql-user=root --mysql-db=workload --tables=32 --table-size=100000 --create_secondary=off --debug=true --threads=32 --mysql-ignore-errors=2013,1213,1105,1205,8022,8027,8028,9004,9007,1062 oltp_write_only prepare

5. Run sysbench and at the same time, do cdc scale and owner switch (Scale CDC from 6 -> 1 -> 5 -> 2)

sysbench --db-driver=mysql --mysql-host=`nslookup upstream-tidb.cdc-testbed-tps-1687429-1-63 | awk -F: '{print $2}' | awk 'NR==5' | sed s/[[:space:]]//g`  --mysql-port=4000 --mysql-user=root --mysql-db=workload --tables=32 --table-size=100000 --create_secondary=off --time=1800 --debug=true --threads=32 --mysql-ignore-errors=2013,1213,1105,1205,8022,8027,8028,9004,9007,1062 oltp_write_only run

7. Wait data sync to consumer target, and do data consistency check

What did you expect to see?

Data should be consistent

What did you see instead?

Data inconsistent

Versions of the cluster

bash-5.1# /cdc version
Release Version: v6.5.2
Git Commit Hash: 1f3151f933c2243af3832177c1356ca842a4c3a7
Git Branch: heads/refs/tags/v6.5.2
UTC Build Time: 2023-04-12 13:56:59
Go Version: go version go1.19.8 linux/amd64
Failpoint Build: false

[fubinzh]

cdc log:
cdc-1687429-0.log
cdc-1687429-1.log

[fubinzh]

/found automation

[fubinzh]

/severity major

[nongfushanquan]

/label affects-6.5

[fubinzh]

/remove-severity major

[fubinzh]

/severity critical

[nongfushanquan]

/close

[ti-chi-bot]

@nongfushanquan: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.