Optimistic sharding may not get initial table structure

[lance6716]

found in CI https://github.com/pingcap/dm/pull/1959/checks?check_run_id=3244290635

in the test case, two worker did INSERT ... VALUES() and ALTER TABLE ... ADD COLUMN in optimistic sharding. one worker has done ADD COLUMN, then another worker starts to get table structure

DM-master's log:

[2021/08/04 17:46:31.830 +00:00] [INFO] [optimist.go:660] ["recorded the initial schema"] [component="shard DDL optimist"] [info="{\"task\":\"upgrade_via_tiup_optimistic\",\"source\":\"mysql-replica-01\",\"up-schema\":\"opt_sharding1\",\"up-table\":\"t1\",\"down-schema\":\"opt_db_target\",\"down-table\":\"t_target\",\"ddls\":[\"ALTER TABLE `opt_db_target`.`t_target` ADD COLUMN `c3` INT\"],\"table-before\":\"CREATE TABLE `tbl`(`c1` INT(11) NOT NULL, `c2` TEXT(65535) CHARACTER SET UTF8MB4 COLLATE utf8mb4_bin, PRIMARY KEY (`c1`)) CHARSET UTF8MB4 COLLATE UTF8MB4_BIN\",\"table-after\":\"CREATE TABLE `tbl`(`c1` INT(11) NOT NULL, `c2` TEXT(65535) CHARACTER SET UTF8MB4 COLLATE utf8mb4_bin, `c3` INT(11), PRIMARY KEY (`c1`)) CHARSET UTF8MB4 COLLATE UTF8MB4_BIN\",\"is-deleted\":false,\"version\":1,\"revision\":30,\"ignore-conflict\":false}"]

DM-worker's log:

[2021/08/04 17:46:31.694 +00:00] [INFO] [optimist.go:175] ["no init schema exists"] [task=upgrade_via_tiup_optimistic] [unit="binlog replication"] [component="shard DDL optimist"] [schema=opt_db_target] [table=t_target] [revision=29]
[2021/08/04 17:46:31.828 +00:00] [INFO] [optimist.go:175] ["no init schema exists"] [task=upgrade_via_tiup_optimistic] [unit="binlog replication"] [component="shard DDL optimist"] [schema=opt_db_target] [table=t_target] [revision=30]
... 
[2021/08/04 17:46:31.849 +00:00] [WARN] [db.go:169] ["execute statements"] [task=upgrade_via_tiup_optimistic] [unit="binlog replication"] [retry=0] [queries="[REPLACE INTO `opt_db_target`.`t_target` (`c1`,`c2`) VALUES (?,?)]"] [arguments="[[111 [49 49 49]]]"] [error="[code=10006:class=database:scope=not-set:level=high], Message: execute statement failed: commit, RawCause: Error 8028: Information schema is changed during the execution of the statement(for example, table definition may be updated by other DDL ran in parallel). If you see this error often, try increasing `tidb_max_delta_schema_count`. [try again later]"]
...
... gen insert sqls failed, schema: opt_db_target, table: t_target: Column count doesn't match value count: 3 (columns) vs 2 (values)

the latter worker read table structure from downstream, so it fails with "Column count doesn't match value count" error because downstream has finished ADD COLUMN