server: only owner do bootstrap #10029

crazycs520 · 2019-04-03T11:35:39Z

What problem does this PR solve?

implement issue: #10017
When there multiple tidb-server do bootstrap, there will be multiple conflict when create system table.
Below is result when I start 5 tidb-server at the same time:

test > admin show ddl jobs 50;
+--------+---------+----------------------+---------------+--------------+-----------+----------+-----------+-----------------------------------+-----------+
| JOB_ID | DB_NAME | TABLE_NAME           | JOB_TYPE      | SCHEMA_STATE | SCHEMA_ID | TABLE_ID | ROW_COUNT | START_TIME                        | STATE     |
+--------+---------+----------------------+---------------+--------------+-----------+----------+-----------+-----------------------------------+-----------+
| 68     | mysql   | bind_info            | create table  | public       | 11        | 67       | 0         | 2019-04-03 19:18:27.193 +0800 CST | synced    |
| 66     | mysql   | default_roles        | create table  | public       | 11        | 65       | 0         | 2019-04-03 19:18:27.141 +0800 CST | synced    |
| 64     | mysql   | role_edges           | create table  | public       | 11        | 63       | 0         | 2019-04-03 19:18:27.04 +0800 CST  | synced    |
| 62     | mysql   |                      | create table  | none         | 11        | 61       | 0         | 2019-04-03 19:18:26.991 +0800 CST | cancelled |
| 60     | mysql   |                      | create table  | none         | 11        | 59       | 0         | 2019-04-03 19:18:26.939 +0800 CST | cancelled |
| 58     | mysql   |                      | create table  | none         | 11        | 57       | 0         | 2019-04-03 19:18:26.892 +0800 CST | cancelled |
| 56     | mysql   | stats_feedback       | create table  | public       | 11        | 55       | 0         | 2019-04-03 19:18:26.842 +0800 CST | synced    |
| 54     | mysql   | gc_delete_range_done | create table  | public       | 11        | 53       | 0         | 2019-04-03 19:18:26.842 +0800 CST | synced    |
| 52     | mysql   |                      | create table  | none         | 11        | 51       | 0         | 2019-04-03 19:18:26.789 +0800 CST | cancelled |
| 50     | mysql   | gc_delete_range      | create table  | public       | 11        | 49       | 0         | 2019-04-03 19:18:26.739 +0800 CST | synced    |
| 48     | mysql   | stats_buckets        | create table  | public       | 11        | 47       | 0         | 2019-04-03 19:18:26.689 +0800 CST | synced    |
| 46     | mysql   | stats_histograms     | create table  | public       | 11        | 45       | 0         | 2019-04-03 19:18:26.591 +0800 CST | synced    |
| 44     | mysql   | stats_meta           | create table  | public       | 11        | 43       | 0         | 2019-04-03 19:18:26.539 +0800 CST | synced    |
| 42     | mysql   | help_topic           | create table  | public       | 11        | 41       | 0         | 2019-04-03 19:18:26.491 +0800 CST | synced    |
| 40     | mysql   |                      | create table  | none         | 11        | 39       | 0         | 2019-04-03 19:18:26.391 +0800 CST | cancelled |
| 38     | mysql   |                      | create table  | none         | 11        | 37       | 0         | 2019-04-03 19:18:26.391 +0800 CST | cancelled |
| 36     | mysql   |                      | create table  | none         | 11        | 35       | 0         | 2019-04-03 19:18:26.391 +0800 CST | cancelled |
| 34     | mysql   | tidb                 | create table  | public       | 11        | 33       | 0         | 2019-04-03 19:18:26.339 +0800 CST | synced    |
| 32     | mysql   | GLOBAL_VARIABLES     | create table  | public       | 11        | 31       | 0         | 2019-04-03 19:18:26.339 +0800 CST | synced    |
| 30     | mysql   |                      | create table  | none         | 11        | 29       | 0         | 2019-04-03 19:18:26.291 +0800 CST | cancelled |
| 28     | mysql   | columns_priv         | create table  | public       | 11        | 27       | 0         | 2019-04-03 19:18:26.242 +0800 CST | synced    |
| 26     | mysql   | tables_priv          | create table  | public       | 11        | 25       | 0         | 2019-04-03 19:18:26.19 +0800 CST  | synced    |
| 24     | mysql   | db                   | create table  | public       | 11        | 23       | 0         | 2019-04-03 19:18:26.142 +0800 CST | synced    |
| 22     | mysql   |                      | create table  | none         | 11        | 21       | 0         | 2019-04-03 19:18:25.894 +0800 CST | cancelled |
| 20     | mysql   |                      | create table  | none         | 11        | 19       | 0         | 2019-04-03 19:18:25.894 +0800 CST | cancelled |
| 18     | mysql   |                      | create table  | none         | 11        | 17       | 0         | 2019-04-03 19:18:25.842 +0800 CST | cancelled |
| 16     | mysql   |                      | create table  | none         | 11        | 15       | 0         | 2019-04-03 19:18:25.694 +0800 CST | cancelled |
| 14     | mysql   | user                 | create table  | public       | 11        | 13       | 0         | 2019-04-03 19:18:25.54 +0800 CST  | synced    |
| 12     | mysql   |                      | create schema | public       | 11        | 0        | 0         | 2019-04-03 19:18:25.494 +0800 CST | synced    |
| 10     |         |                      | create schema | none         | 9         | 0        | 0         | 2019-04-03 19:18:25.39 +0800 CST  | cancelled |
| 8      |         |                      | create schema | none         | 3         | 0        | 0         | 2019-04-03 19:18:25.39 +0800 CST  | cancelled |
| 7      |         |                      | create schema | none         | 1         | 0        | 0         | 2019-04-03 19:18:25.39 +0800 CST  | cancelled |
| 6      |         |                      | create schema | none         | 2         | 0        | 0         | 2019-04-03 19:18:25.339 +0800 CST | cancelled |
| 5      | test    |                      | create schema | public       | 4         | 0        | 0         | 2019-04-03 19:18:25.189 +0800 CST | synced    |
+--------+---------+----------------------+---------------+--------------+-----------+----------+-----------+-----------------------------------+-----------+

As you can see, there are multiple DDL-job was been canceled.

Actual only 1 tidb-server need to do bootstrap to create system table.

So which one? The DDL-owner

What is changed and how it works?

Add a judge to check whether is owner? If is owner, to do bootstrap to create system table.

In this PR, start 5 tidb-server at the same time, there is no conflict.

test> admin show ddl jobs 50;
+--------+---------+----------------------+---------------+--------------+-----------+----------+-----------+-----------------------------------+--------+
| JOB_ID | DB_NAME | TABLE_NAME           | JOB_TYPE      | SCHEMA_STATE | SCHEMA_ID | TABLE_ID | ROW_COUNT | START_TIME                        | STATE  |
+--------+---------+----------------------+---------------+--------------+-----------+----------+-----------+-----------------------------------+--------+
| 36     | mysql   | bind_info            | create table  | public       | 3         | 35       | 0         | 2019-04-03 19:32:42.293 +0800 CST | synced |
| 34     | mysql   | default_roles        | create table  | public       | 3         | 33       | 0         | 2019-04-03 19:32:42.242 +0800 CST | synced |
| 32     | mysql   | role_edges           | create table  | public       | 3         | 31       | 0         | 2019-04-03 19:32:42.142 +0800 CST | synced |
| 30     | mysql   | stats_feedback       | create table  | public       | 3         | 29       | 0         | 2019-04-03 19:32:42.092 +0800 CST | synced |
| 28     | mysql   | gc_delete_range_done | create table  | public       | 3         | 27       | 0         | 2019-04-03 19:32:42.042 +0800 CST | synced |
| 26     | mysql   | gc_delete_range      | create table  | public       | 3         | 25       | 0         | 2019-04-03 19:32:41.942 +0800 CST | synced |
| 24     | mysql   | stats_buckets        | create table  | public       | 3         | 23       | 0         | 2019-04-03 19:32:41.893 +0800 CST | synced |
| 22     | mysql   | stats_histograms     | create table  | public       | 3         | 21       | 0         | 2019-04-03 19:32:41.843 +0800 CST | synced |
| 20     | mysql   | stats_meta           | create table  | public       | 3         | 19       | 0         | 2019-04-03 19:32:41.742 +0800 CST | synced |
| 18     | mysql   | help_topic           | create table  | public       | 3         | 17       | 0         | 2019-04-03 19:32:41.691 +0800 CST | synced |
| 16     | mysql   | tidb                 | create table  | public       | 3         | 15       | 0         | 2019-04-03 19:32:41.643 +0800 CST | synced |
| 14     | mysql   | GLOBAL_VARIABLES     | create table  | public       | 3         | 13       | 0         | 2019-04-03 19:32:41.542 +0800 CST | synced |
| 12     | mysql   | columns_priv         | create table  | public       | 3         | 11       | 0         | 2019-04-03 19:32:41.493 +0800 CST | synced |
| 10     | mysql   | tables_priv          | create table  | public       | 3         | 9        | 0         | 2019-04-03 19:32:41.443 +0800 CST | synced |
| 8      | mysql   | db                   | create table  | public       | 3         | 7        | 0         | 2019-04-03 19:32:41.343 +0800 CST | synced |
| 6      | mysql   | user                 | create table  | public       | 3         | 5        | 0         | 2019-04-03 19:32:41.294 +0800 CST | synced |
| 4      | mysql   |                      | create schema | public       | 3         | 0        | 0         | 2019-04-03 19:32:41.242 +0800 CST | synced |
| 2      | test    |                      | create schema | public       | 1         | 0        | 0         | 2019-04-03 19:32:41.141 +0800 CST | synced |
+--------+---------+----------------------+---------------+--------------+-----------+----------+-----------+-----------------------------------+--------+

Check List

Tests

I don't know how to test? Any advise please let me know?

Code changes

Side effects

Related changes

crazycs520 · 2019-04-03T11:47:16Z

@winkyao @zimulala @tiancaiamao

codecov · 2019-04-03T11:52:02Z

Codecov Report

❗ No coverage uploaded for pull request base (master@e90f3b9). Click here to learn what that means.
The diff coverage is 50%.

@@             Coverage Diff             @@
##             master     #10029   +/-   ##
===========================================
  Coverage          ?   78.0309%           
===========================================
  Files             ?        405           
  Lines             ?      82161           
  Branches          ?          0           
===========================================
  Hits              ?      64111           
  Misses            ?      13342           
  Partials          ?       4708

session/bootstrap.go

xiekeyi98

Do we need some logs here?
Such as, logutil.Logger(context.Background()).Info("bootstrap success");

session/bootstrap.go

tiancaiamao · 2019-04-08T09:10:36Z

LGTM

… into bootstrap-in-owner

session/bootstrap.go

crazycs520 · 2019-04-09T12:32:53Z

@xiekeyi98 After consideration, I think print "bootstrap success" log here is too early, there is a lot of other jobs need to do. Maybe we can add this log in some other place.

winkyao

LGTM

winkyao · 2019-04-10T06:50:03Z

@zimulala PTAL

zimulala

LGTM

… into bootstrap-in-owner

crazycs520 · 2019-04-10T11:09:53Z

/run-all-tests

crazycs520 · 2019-04-10T11:10:15Z

/run-all-tests

zimulala · 2019-04-10T13:20:13Z

/run-all-tests

shenli · 2019-04-10T13:22:02Z

The cancelled jobs have only a few info. Is it a bug?

40     | mysql   |                      | create table  | none         | 11        | 39       | 0         | 2019-04-03 19:18:26.391 +0800 CST | cancelled |

crazycs520 · 2019-04-11T03:04:11Z

Because the table name is got from domin.infoschema by table ID, but the create table job was cancelled, so can't find table by job.tableID.

Maybe we can get the table name from job.Arg.

server: only owner do bootstrap

6211344

crazycs520 added component/server require-LGT3 Indicates that the PR requires three LGTM. labels Apr 3, 2019

crazycs520 requested a review from tiancaiamao April 3, 2019 11:46

Merge branch 'master' into bootstrap-in-owner

ff56efe

winkyao reviewed Apr 4, 2019

View reviewed changes

session/bootstrap.go Show resolved Hide resolved

xiekeyi98 reviewed Apr 4, 2019

View reviewed changes

crazycs520 added 2 commits April 8, 2019 11:59

Address comment

7057924

Merge branch 'master' into bootstrap-in-owner

b1f2d6c

zimulala reviewed Apr 8, 2019

View reviewed changes

session/bootstrap.go Show resolved Hide resolved

tiancaiamao added the status/LGT1 Indicates that a PR has LGTM 1. label Apr 8, 2019

crazycs520 added 2 commits April 9, 2019 17:14

address comment

67e3f76

Merge branch 'bootstrap-in-owner' of https://github.com/crazycs520/tidb…

3fb8864

… into bootstrap-in-owner

zimulala reviewed Apr 9, 2019

View reviewed changes

session/bootstrap.go Outdated Show resolved Hide resolved

crazycs520 added 2 commits April 10, 2019 14:41

enlarge sleep time when bootstrap

ee7fd24

Merge branch 'master' into bootstrap-in-owner

259c084

winkyao reviewed Apr 10, 2019

View reviewed changes

zimulala previously approved these changes Apr 10, 2019

View reviewed changes

crazycs520 added 2 commits April 10, 2019 18:51

fix bug for upgrade

45d052f

Merge branch 'bootstrap-in-owner' of https://github.com/crazycs520/tidb…

a47e6e6

… into bootstrap-in-owner

crazycs520 dismissed zimulala’s stale review via a47e6e6 April 10, 2019 10:53

Merge branch 'master' into bootstrap-in-owner

42306a8

Merge branch 'master' into bootstrap-in-owner

dce035b

zimulala approved these changes Apr 10, 2019

View reviewed changes

zimulala added status/LGT3 The PR has already had 3 LGTM. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Apr 10, 2019

Merge branch 'master' into bootstrap-in-owner

ad1e87a

crazycs520 merged commit 5a34c4b into pingcap:master Apr 11, 2019

crazycs520 deleted the bootstrap-in-owner branch April 11, 2019 03:04

crazycs520 added a commit to crazycs520/tidb that referenced this pull request May 13, 2019

server: only owner do bootstrap (pingcap#10029)

33da90f

crazycs520 mentioned this pull request May 13, 2019

server: only owner do bootstrap (#10029) #10426

Merged

jackysp pushed a commit that referenced this pull request May 13, 2019

server: only owner do bootstrap (#10029) (#10426)

570a315

zimulala mentioned this pull request Jun 6, 2019

Potential race condition in CREATE TABLE #10410

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

server: only owner do bootstrap #10029

server: only owner do bootstrap #10029

crazycs520 commented Apr 3, 2019 •

edited

Loading

crazycs520 commented Apr 3, 2019

codecov bot commented Apr 3, 2019 •

edited

Loading

xiekeyi98 left a comment

tiancaiamao commented Apr 8, 2019

crazycs520 commented Apr 9, 2019

winkyao left a comment

winkyao commented Apr 10, 2019

zimulala left a comment

crazycs520 commented Apr 10, 2019

crazycs520 commented Apr 10, 2019

zimulala commented Apr 10, 2019

shenli commented Apr 10, 2019

crazycs520 commented Apr 11, 2019

server: only owner do bootstrap #10029

server: only owner do bootstrap #10029

Conversation

crazycs520 commented Apr 3, 2019 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

crazycs520 commented Apr 3, 2019

codecov bot commented Apr 3, 2019 • edited Loading

Codecov Report

xiekeyi98 left a comment

Choose a reason for hiding this comment

tiancaiamao commented Apr 8, 2019

crazycs520 commented Apr 9, 2019

winkyao left a comment

Choose a reason for hiding this comment

winkyao commented Apr 10, 2019

zimulala left a comment

Choose a reason for hiding this comment

crazycs520 commented Apr 10, 2019

crazycs520 commented Apr 10, 2019

zimulala commented Apr 10, 2019

shenli commented Apr 10, 2019

crazycs520 commented Apr 11, 2019

crazycs520 commented Apr 3, 2019 •

edited

Loading

codecov bot commented Apr 3, 2019 •

edited

Loading