panic: checkpointTs regressed

[Tammyxia]

What did you do?

• 1 CDC node, create changefeed: tiup cdc:nightly cli changefeed create --pd=http://172.16.6.5:2379 --sink-uri="kafka://172.16.6.47:9092/cdc-events-t1?kafka-version=3.1.0&partition-num=12&protocol=canal-json" --changefeed-id="test-1"
• Run workload: tiup tidb-lightning --config lightning.toml
  [tikv-importer]
  backend = "tidb"
  sorted-kv-dir = "/home/ssd/sort"
  [mydumper]
  data-source-dir = "s3://nfs/lixia-tools-workload/CDC/comp-PK-UKnotnull?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%2fminio.pingcap.net%3a9000&force-path-style=true"

What did you expect to see?

No response

What did you see instead?

panic: checkpointTs regressed

goroutine 1208153 [running]:
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc170ffc0c0, {0xc1f043eb00, 0x2, 0x2})
go.uber.org/zap@v1.21.0/zapcore/entry.go:232 +0x44c
go.uber.org/zap.(*Logger).Panic(0xaaaaaaaaaaaaaa?, {0x31eb26d?, 0xc123080bc8?}, {0xc1f043eb00, 0x2, 0x2})
go.uber.org/zap@v1.21.0/logger.go:230 +0x59
github.com/pingcap/tiflow/cdc/scheduler.(*BaseScheduleDispatcher).Tick(0xc428810880, {0x38d22d0, 0xc526349548}, 0x601e44ad4dc000c, {0xc167dd8000?, 0x212e, 0x0?}, 0xc123080bf8?)
github.com/pingcap/tiflow/cdc/scheduler/schedule_dispatcher.go:184 +0x31f
github.com/pingcap/tiflow/cdc/owner.(*schedulerV2).Tick(0xc00143b9f0, {0x38d22d0, 0xc526349548}, 0xc0cbd2c240, {0xc167dd8000, 0x212e, 0x2400}, 0x40d565?)
github.com/pingcap/tiflow/cdc/owner/scheduler.go:121 +0xa8
github.com/pingcap/tiflow/cdc/owner.(*changefeed).tick(0xc3709ce000, {0x38d22d0, 0xc526349548}, 0xa5b81d2bbd2c?, 0x6310fc0?)
github.com/pingcap/tiflow/cdc/owner/changefeed.go:223 +0x73f
github.com/pingcap/tiflow/cdc/owner.(*changefeed).Tick(0xc3709ce000, {0x38d21e0?, 0xc526349530}, 0x6?, 0xc09369ba67364cbd?)
github.com/pingcap/tiflow/cdc/owner/changefeed.go:126 +0x171
github.com/pingcap/tiflow/cdc/owner.(*ownerImpl).Tick(0xc1c03fcd90, {0x7f0c0104c0f0?, 0xc630970320}, {0x38a7ca8?, 0xc05ec45c80})
github.com/pingcap/tiflow/cdc/owner/owner.go:196 +0x3ed
github.com/pingcap/tiflow/pkg/orchestrator.(*EtcdWorker).Run(0xc900881080, {0x7f0c0104c0f0, 0xc630970320}, 0xc830a290e0, 0xbebc200, {0x31ae34a, 0x5})
github.com/pingcap/tiflow/pkg/orchestrator/etcd_worker.go:238 +0xdc3
github.com/pingcap/tiflow/cdc/capture.(*Capture).runEtcdWorker(0xc00033a2a0, {0x38d2280, 0xc630970320}, {0x7f0c01108378, 0xc1c03fcd90}, {0x38a7ca8, 0xc05ec45c80}, 0x191b3ec?, {0x31ae34a, 0x5})
github.com/pingcap/tiflow/cdc/capture/capture.go:463 +0xff
github.com/pingcap/tiflow/cdc/capture.(*Capture).campaignOwner(0xc00033a2a0, {0x38d2280?, 0xc6a9ae98e0?})
github.com/pingcap/tiflow/cdc/capture/capture.go:436 +0x7a8
github.com/pingcap/tiflow/cdc/capture.(*Capture).run.func2()
github.com/pingcap/tiflow/cdc/capture/capture.go:300 +0xa5
created by github.com/pingcap/tiflow/cdc/capture.(*Capture).run
github.com/pingcap/tiflow/cdc/capture/capture.go:294 +0x410
panic: checkpointTs regressed
...

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)
6.1.0 nightly

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)
TiKV
Release Version:   6.0.0-alpha
Edition:           Community
Git Commit Hash:   c3d09adcc2bc25b79d3fff053e66a16420d3bce7
Git Commit Branch: heads/refs/tags/v6.1.0-alpha
UTC Build Time:    2022-04-27 06:35:47
Rust Version:      rustc 1.60.0-nightly (1e12aef3f 2022-02-13)
Enable Features:   jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure
Profile:           dist_release

TiCDC version (execute cdc version):

(paste TiCDC version here)
./cdc version
Release Version: v6.1.0-alpha
Git Commit Hash: 8cbf453552b43b81d135d8b12066fc4a4b281591
Git Branch: heads/refs/tags/v6.1.0-alpha
UTC Build Time: 2022-04-27 14:52:50
Go Version: go version go1.18 linux/amd64
Failpoint Build: false

[cyliu0]

/assign @nongfushanquan

[sdojjy]

panic log

[2022/04/30 22:01:21.151 +08:00] [PANIC] [schedule_dispatcher.go:184] ["checkpointTs regressed"] [changefeed=test-1] [old=432878070710665228] [new=432878070697820173] [stack="github.com/pingcap/tiflow/cdc/scheduler.(*BaseScheduleDispatcher).Tick\n\tgithub.com/pingcap/tiflow/cdc/scheduler/schedule_dispatcher.go:184\ngithub.com/pingcap/tiflow/cdc/owner.(*schedulerV2).Tick\n\tgithub.com/pingcap/tiflow/cdc/owner/scheduler.go:121\ngithub.com/pingcap/tiflow/cdc/owner.(*changefeed).tick\n\tgithub.com/pingcap/tiflow/cdc/owner/changefeed.go:223\ngithub.com/pingcap/tiflow/cdc/owner.(*changefeed).Tick\n\tgithub.com/pingcap/tiflow/cdc/owner/changefeed.go:126\ngithub.com/pingcap/tiflow/cdc/owner.(*ownerImpl).Tick\n\tgithub.com/pingcap/tiflow/cdc/owner/owner.go:196\ngithub.com/pingcap/tiflow/pkg/orchestrator.(*EtcdWorker).Run\n\tgithub.com/pingcap/tiflow/pkg/orchestrator/etcd_worker.go:238\ngithub.com/pingcap/tiflow/cdc/capture.(*Capture).runEtcdWorker\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:463\ngithub.com/pingcap/tiflow/cdc/capture.(*Capture).campaignOwner\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:436\ngithub.com/pingcap/tiflow/cdc/capture.(*Capture).run.func2\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:300"]
[2022/04/30 23:24:21.100 +08:00] [PANIC] [schedule_dispatcher.go:184] ["checkpointTs regressed"] [changefeed=test-1] [old=432878094264303627] [new=432878094238351373] [stack="github.com/pingcap/tiflow/cdc/scheduler.(*BaseScheduleDispatcher).Tick\n\tgithub.com/pingcap/tiflow/cdc/scheduler/schedule_dispatcher.go:184\ngithub.com/pingcap/tiflow/cdc/owner.(*schedulerV2).Tick\n\tgithub.com/pingcap/tiflow/cdc/owner/scheduler.go:121\ngithub.com/pingcap/tiflow/cdc/owner.(*changefeed).tick\n\tgithub.com/pingcap/tiflow/cdc/owner/changefeed.go:223\ngithub.com/pingcap/tiflow/cdc/owner.(*changefeed).Tick\n\tgithub.com/pingcap/tiflow/cdc/owner/changefeed.go:126\ngithub.com/pingcap/tiflow/cdc/owner.(*ownerImpl).Tick\n\tgithub.com/pingcap/tiflow/cdc/owner/owner.go:196\ngithub.com/pingcap/tiflow/pkg/orchestrator.(*EtcdWorker).Run\n\tgithub.com/pingcap/tiflow/pkg/orchestrator/etcd_worker.go:238\ngithub.com/pingcap/tiflow/cdc/capture.(*Capture).runEtcdWorker\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:463\ngithub.com/pingcap/tiflow/cdc/capture.(*Capture).campaignOwner\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:436\ngithub.com/pingcap/tiflow/cdc/capture.(*Capture).run.func2\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:300"]