DM integration test full_mode is unstable.

[asddongmen]

Which jobs are flaking?

https://ci2.pingcap.net/blue/organizations/jenkins/dm_ghpr_integration_test/detail/dm_ghpr_integration_test/2599/pipeline/

Which test(s) are flaking?

full_mode

Jenkins logs or GitHub Actions link

[2022-01-10T03:50:26.977Z] /home/jenkins/agent/workspace/dm_ghpr_integration_test/go/src/github.com/pingcap/tiflow/dm/bin/dm-syncer.test --log-file=/tmp/dm_test/dm_syncer/syncer1/log/dm-syncer.log --config=/tmp/dm_test/dm_syncer/dm-syncer-1.toml   >> /tmp/dm_test/dm_syncer/syncer1/log/stdout.log 2>&1 &

[2022-01-10T03:50:26.977Z] /home/jenkins/agent/workspace/dm_ghpr_integration_test/go/src/github.com/pingcap/tiflow/dm/bin/dm-syncer.test --log-file=/tmp/dm_test/dm_syncer/syncer2/log/dm-syncer.log --config=/tmp/dm_test/dm_syncer/dm-syncer-2.toml --meta=/tmp/dm_test/dm_syncer/old_meta_file --syncer-config-format >> /tmp/dm_test/dm_syncer/syncer2/log/stdout.log 2>&1 &

[2022-01-10T03:50:36.996Z] check diff successfully

[2022-01-10T03:50:37.261Z] check diff successfully

[2022-01-10T03:50:37.261Z] check diff failed 1-th time, retry later

[2022-01-10T03:50:39.801Z] check diff failed 2-th time, retry later

[2022-01-10T03:50:41.705Z] check diff failed 3-th time, retry later

[2022-01-10T03:50:44.245Z] check diff failed 4-th time, retry later

[2022-01-10T03:50:46.149Z] check diff failed 5-th time, retry later

[2022-01-10T03:50:48.081Z] check diff failed 6-th time, retry later

[2022-01-10T03:50:50.605Z] check diff failed 7-th time, retry later

[2022-01-10T03:50:52.507Z] check diff failed 8-th time, retry later

[2022-01-10T03:50:54.410Z] check diff failed 9-th time, retry later

[2022-01-10T03:50:56.946Z] check diff failed 10-th time, retry later

[2022-01-10T03:50:58.845Z] check diff failed at last

[2022-01-10T03:50:58.845Z] There is something error when initialize diff, please check log info in /tmp/ticdc_dm_test/output/sync_diff.log

[2022-01-10T03:50:58.845Z] 

[2022-01-10T03:50:58.845Z] [2022/01/10 11:50:56.448 +08:00] [INFO] [printer.go:46] ["Welcome to sync_diff_inspector"] ["Release Version"=v5.4.0] ["Git Commit Hash"=b1def7746ec382a71a70b8958b1053202092f95c] ["Git Branch"=heads/refs/tags/v5.4.0] ["UTC Build Time"="2022-01-03 10:01:35"] ["Go Version"=go1.16.4]

[2022-01-10T03:50:58.845Z] [2022/01/10 11:50:56.449 +08:00] [INFO] [main.go:93] [config="{\"check-thread-count\":4,\"export-fix-sql\":true,\"check-struct-only\":false,\"dm-addr\":\"\",\"dm-task\":\"\",\"data-sources\":{\"mysql1\":{\"host\":\"127.0.0.1\",\"port\":3307,\"user\":\"root\",\"password\":\"123456\",\"sql-mode\":\"\",\"snapshot\":\"\",\"route-rules\":[\"rule1\"],\"Router\":{\"Selector\":{}},\"Conn\":null},\"tidb0\":{\"host\":\"127.0.0.1\",\"port\":4000,\"user\":\"test\",\"password\":\"123456\",\"sql-mode\":\"\",\"snapshot\":\"\",\"route-rules\":null,\"Router\":{\"Selector\":{}},\"Conn\":null}},\"routes\":{\"rule1\":{\"schema-pattern\":\"dm_syncer\",\"table-pattern\":\"dm_syncer_route_rules_*\",\"target-schema\":\"dm_syncer\",\"target-table\":\"dm_syncer_route_rule\"}},\"table-configs\":null,\"task\":{\"source-instances\":[\"mysql1\"],\"source-routes\":null,\"target-instance\":\"tidb0\",\"target-check-tables\":[\"dm_syncer.dm_syncer_route_rule\"],\"target-configs\":null,\"output-dir\":\"/tmp/ticdc_dm_test/output\",\"SourceInstances\":[{\"host\":\"127.0.0.1\",\"port\":3307,\"user\":\"root\",\"password\":\"123456\",\"sql-mode\":\"\",\"snapshot\":\"\",\"route-rules\":[\"rule1\"],\"Router\":{\"Selector\":{}},\"Conn\":null}],\"TargetInstance\":{\"host\":\"127.0.0.1\",\"port\":4000,\"user\":\"test\",\"password\":\"123456\",\"sql-mode\":\"\",\"snapshot\":\"\",\"route-rules\":null,\"Router\":{\"Selector\":{}},\"Conn\":null},\"TargetTableConfigs\":null,\"TargetCheckTables\":[{}],\"FixDir\":\"/tmp/ticdc_dm_test/output/fix-on-tidb0\",\"CheckpointDir\":\"/tmp/ticdc_dm_test/output/checkpoint\",\"HashFile\":\"\"},\"ConfigFile\":\"/home/jenkins/agent/workspace/dm_ghpr_integration_test/go/src/github.com/pingcap/tiflow/dm/tests/dm_syncer/conf/diff_config_route_rules.toml\",\"PrintVersion\":false}"]

[2022-01-10T03:50:58.845Z] [2022/01/10 11:50:56.451 +08:00] [INFO] [diff.go:826] ["set tidb cfg"]

[2022-01-10T03:50:58.846Z] [2022/01/10 11:50:56.457 +08:00] [FATAL] [main.go:112] ["failed to initialize diff process"] [error="no table need to be compared"] [errorVerbose="no table need to be compared\ngithub.com/pingcap/tidb-tools/sync_diff_inspector/source.NewSources\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-tools/sync_diff_inspector/source/source.go:149\nmain.(*Diff).init\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-tools/sync_diff_inspector/diff.go:139\nmain.NewDiff\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-tools/sync_diff_inspector/diff.go:97\nmain.checkSyncState\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-tools/sync_diff_inspector/main.go:109\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-tools/sync_diff_inspector/main.go:96\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"] [stack="main.checkSyncState\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-tools/sync_diff_inspector/main.go:112\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-tools/sync_diff_inspector/main.go:96\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"]

[2022-01-10T03:50:58.846Z] 

[2022-01-10T03:50:58.846Z] ...

[2022-01-10T03:50:58.846Z] make: *** [dm_integration_test] Error 1

script returned exit code 2

https://ci2.pingcap.net/blue/organizations/jenkins/dm_ghpr_integration_test/detail/dm_ghpr_integration_test/2599/pipeline/

Anything else we need to know

• Does this test exist for other branches as well?

• Has there been a high frequency of failure lately?

[asddongmen]

it never orrurs