"fail to initial unit Sync of subtask" with many do tables

[csuzhangxc]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

   start-task

2. What did you expect to see?

   no errors

3. What did you see instead?

4. Versions of the cluster

   • DM version (run dmctl -V or dm-worker -V or dm-master -V):

     v2.0.0
     

goroutines:

[csuzhangxc]

the flow of init Sync unit checkpoint:

1. read a row of checkpoint (for a database or table)
2. track table info in schema-tracker
3. read the next row until no more rows exist

maybe the cause:

if many tables exist AND tracking each row takes a little long time, the total time will become very long, then deadline exceeded.

[aeoluswing]

Has this issue already fixed? I also encountered the same error in production mode(version 2.0.0).

[lance6716]

Has this issue already fixed? I also encountered the same error in production mode(version 2.0.0).

fixed in nightly but not v2.0.0. You could find https://github.com/pingcap/dm/releases/tag/v2.0.0-hotfix for v2.0.0 or ask help in AskTUG.com

[aeoluswing]

Sorry,this hotfix package has a path error:

tidb@Monitor-01:~/data/dm_master$ tiup dm display dm-cluster
Starting component `dm`: /home/tidb/.tiup/components/dm/v1.2.4/tiup-dm display dm-cluster
Cluster type:    dm
Cluster name:    dm-cluster
Cluster version: v2.0.0
SSH type:        builtin
ID                  Role          Host           Ports      OS/Arch       Status     Data Dir                                                              Deploy Dir
--                  ----          ----           -----      -------       ------     --------                                                              ----------
10.66.193.60:9193   alertmanager  10.66.193.60   9193/9094  linux/x86_64  Up         /home/tidb/data/dm_monitor/deploy/alertmanager/data.alertmanager      /home/tidb/data/dm_monitor/deploy/alertmanager/alertmanager-9193
10.66.193.60:8261   dm-master     10.66.193.60   8261/8291  linux/x86_64  Healthy|L  /home/tidb/data/dm_master/deploy/dm-master-8261/data                  /home/tidb/data/dm_master/deploy/dm-master-8261
10.66.193.58:8262   dm-worker     10.66.193.58   8262       linux/x86_64  Bound      /home/tidb/data/dm_worker/deploy/data                                 /home/tidb/data/dm_worker/deploy
10.66.193.59:8262   dm-worker     10.66.193.59   8262       linux/x86_64  Bound      /home/tidb/data/dm_worker/deploy/data                                 /home/tidb/data/dm_worker/deploy
10.66.205.211:8262  dm-worker     10.66.205.211  8262       linux/x86_64  Bound      /home/tidb/data/dm_worker/deploy/data                                 /home/tidb/data/dm_worker/deploy
10.66.193.60:9000   grafana       10.66.193.60   9000       linux/x86_64  Up         -                                                                     /home/tidb/data/dm_monitor/deploy/grafana/grafana-9000
10.66.193.60:9190   prometheus    10.66.193.60   9190       linux/x86_64  Up         /home/tidb/data/dm_monitor/deploy/prometheus/prometheus.data.metrics  /home/tidb/data/dm_monitor/deploy/prometheus/prometheus-9190
Total nodes: 7
tidb@Monitor-01:~/data/dm_master$ tiup dm patch dm-cluster ./dm-2.0.0-hotfix.tar.gz -R dm-master
Starting component `dm`: /home/tidb/.tiup/components/dm/v1.2.4/tiup-dm patch dm-cluster ./dm-2.0.0-hotfix.tar.gz -R dm-master

Error: entry dm-master/dm-master not found in package ./dm-2.0.0-hotfix.tar.gz

Verbose debug logs has been written to /home/tidb/data/dm_master/logs/tiup-cluster-debug-2020-12-04-20-33-00.log.
Error: run `/home/tidb/.tiup/components/dm/v1.2.4/tiup-dm` (wd:/home/tidb/.tiup/data/SICjdXB) failed: exit status 1
tidb@Monitor-01:~/data/dm_master$ tiup dm patch dm-cluster dm-2.0.0-hotfix.tar.gz -N 10.66.193.60:8261
Starting component `dm`: /home/tidb/.tiup/components/dm/v1.2.4/tiup-dm patch dm-cluster dm-2.0.0-hotfix.tar.gz -N 10.66.193.60:8261

Error: entry dm-master/dm-master not found in package dm-2.0.0-hotfix.tar.gz

Verbose debug logs has been written to /home/tidb/data/dm_master/logs/tiup-cluster-debug-2020-12-04-20-33-16.log.
Error: run `/home/tidb/.tiup/components/dm/v1.2.4/tiup-dm` (wd:/home/tidb/.tiup/data/SICjhhQ) failed: exit status 1

[aeoluswing]

@lance6716 asktug.com/t/topic/63818/4 has a similar error.My environment is a private cloud,so command upgrade nightly encountered a network error.

[aeoluswing]

I found that the package name is not consistent with the download link.so ... I guess it's not the correct package.

[lance6716]

I found that the package name is not consistent with the download link.so ... I guess it's not the correct package.

you could replace binaries from GitHub release manually, or refer to doc to do a offline nightly TiUP deployment, or request help in asktug.com

[lance6716]

https://docs.pingcap.com/zh/tidb-data-migration/stable/deploy-a-dm-cluster-using-tiup-offline

[aeoluswing]

@lance6716 as you said,update tiup and dm version to nightly could solove this issue.Thx a lot.