test(dm): drop_column_with_index is unstable

[okJiang]

Which jobs are flaking?

https://ci2.pingcap.net/blue/organizations/jenkins/dm_ghpr_integration_test/detail/dm_ghpr_integration_test/2565/pipeline/

Which test(s) are flaking?

drop_column_with_index

Jenkins logs or GitHub Actions link

[2022-01-07T08:27:42.536Z] start running case: [drop_column_with_index] script: [/home/jenkins/agent/workspace/dm_ghpr_integration_test/go/src/github.com/pingcap/tiflow/dm/tests/drop_column_with_index/run.sh]

[2022-01-07T08:27:42.536Z] Running test /home/jenkins/agent/workspace/dm_ghpr_integration_test/go/src/github.com/pingcap/tiflow/dm/tests/drop_column_with_index/run.sh...

[2022-01-07T08:27:42.536Z] Verbose mode = false

[2022-01-07T08:27:42.536Z] 0 dm-master alive

[2022-01-07T08:27:42.536Z] 0 dm-worker alive

[2022-01-07T08:27:42.536Z] 0 dm-syncer alive

[2022-01-07T08:27:42.536Z] process dm-master.test already exit

[2022-01-07T08:27:42.536Z] process dm-worker.test already exit

[2022-01-07T08:27:42.536Z] process dm-syncer.test already exit

[2022-01-07T08:27:42.536Z] [Fri Jan  7 16:27:42 CST 2022] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/dm_ghpr_integration_test/go/src/github.com/pingcap/tiflow/dm/tests/drop_column_with_index/conf/dm-master.toml >>>>>>

[2022-01-07T08:27:42.536Z] wait for rpc addr 127.0.0.1:8261 alive the 1-th time

[2022-01-07T08:27:43.465Z] rpc addr 127.0.0.1:8261 is alive

[2022-01-07T08:27:43.465Z] [Fri Jan  7 16:27:43 CST 2022] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/dm_ghpr_integration_test/go/src/github.com/pingcap/tiflow/dm/tests/drop_column_with_index/conf/dm-worker1.toml >>>>>>

[2022-01-07T08:27:43.465Z] wait for rpc addr 127.0.0.1:8262 alive the 1-th time

[2022-01-07T08:27:44.833Z] rpc addr 127.0.0.1:8262 is alive

[2022-01-07T08:27:44.833Z] dmctl test cmd: "operate-source create /tmp/dm_test/drop_column_with_index/source1.yaml"

[2022-01-07T08:27:45.761Z] dmctl test cmd: "start-task /home/jenkins/agent/workspace/dm_ghpr_integration_test/go/src/github.com/pingcap/tiflow/dm/tests/drop_column_with_index/conf/dm-task.yaml --remove-meta"

[2022-01-07T08:27:47.659Z] dmctl test cmd: "query-status test"

[2022-01-07T08:27:47.659Z] got=1 expected=1

[2022-01-07T08:27:47.659Z] check diff successfully

[2022-01-07T08:27:47.659Z] dmctl test cmd: "query-status test"

[2022-01-07T08:27:47.659Z] got=1 expected=1

[2022-01-07T08:27:47.659Z] got=1 expected=1

[2022-01-07T08:27:47.659Z] got=1 expected=1

[2022-01-07T08:27:47.659Z] dmctl test cmd: "pause-task test"

[2022-01-07T08:27:47.917Z] dmctl test cmd: "query-status test"

[2022-01-07T08:27:47.917Z] got=1 expected=1

[2022-01-07T08:27:47.917Z] got=1 expected=1

[2022-01-07T08:27:47.917Z] got=1 expected=1

[2022-01-07T08:27:53.165Z] grep: dispatch: No such file or directory

[2022-01-07T08:27:53.165Z] grep: auto: No such file or directory

[2022-01-07T08:27:53.165Z] grep: resume: No such file or directory

[2022-01-07T08:27:53.165Z] grep: task: No such file or directory

[2022-01-07T08:27:53.166Z] dmctl test cmd: "resume-task test"

[2022-01-07T08:27:53.422Z] check diff failed 1-th time, retry later

[2022-01-07T08:27:55.942Z] check diff failed 2-th time, retry later

[2022-01-07T08:27:57.831Z] check diff failed 3-th time, retry later

[2022-01-07T08:27:59.721Z] check diff failed 4-th time, retry later

[2022-01-07T08:28:02.239Z] check diff failed 5-th time, retry later

[2022-01-07T08:28:04.128Z] check diff failed 6-th time, retry later

[2022-01-07T08:28:06.020Z] check diff failed 7-th time, retry later

[2022-01-07T08:28:08.537Z] check diff failed 8-th time, retry later

[2022-01-07T08:28:10.427Z] check diff failed 9-th time, retry later

[2022-01-07T08:28:12.321Z] check diff failed 10-th time, retry later

[2022-01-07T08:28:14.216Z] check diff failed at last

[2022-01-07T08:28:14.216Z] /tmp/dm_test/drop_column_with_index/sync_diff_stdout.log

[2022-01-07T08:28:14.216Z] A total of 1 tables need to be compared

[2022-01-07T08:28:14.216Z] 

[2022-01-07T08:28:14.216Z] 

[2022-01-07T08:28:14.216Z] Comparing the table structure of ``drop_column_with_index`.`t1`` ... failure

[2022-01-07T08:28:14.216Z] _____________________________________________________________________________

[2022-01-07T08:28:14.216Z] Progress [============================================================>] 100% 0/0

[2022-01-07T08:28:14.216Z] Progress [============================================================>] 100% 0/0

[2022-01-07T08:28:14.216Z] The structure of `drop_column_with_index`.`t1` is not equal, and data-check is skipped

[2022-01-07T08:28:14.216Z] 

[2022-01-07T08:28:14.216Z] The rest of tables are all equal.

[2022-01-07T09:10:30.852Z] [2022/01/07 17:10:07.990 +08:00] [ERROR] [baseconn.go:184] ["execute statement failed"] [task=test] [unit="binlog replication"] [query="ALTER TABLE `drop_column_with_index`.`t1` DROP COLUMN `c1`"] [argument="[]"] [error="Error 8200: can't drop column c1 with tidb_enable_change_multi_schema is disable"]
[2022-01-07T09:10:30.852Z] [2022/01/07 17:10:07.992 +08:00] [ERROR] [db.go:221] ["execute statements failed after retry"] [task=test] [unit="binlog replication"] [queries="[ALTER TABLE `drop_column_with_index`.`t1` DROP COLUMN `c1`]"] [arguments="[]"] [error="[code=10006:class=database:scope=not-set:level=high], Message: execute statement failed: ALTER TABLE `drop_column_with_index`.`t1` DROP COLUMN `c1`, RawCause: Error 8200: can't drop column c1 with tidb_enable_change_multi_schema is disable"]
[2022-01-07T09:10:30.852Z] [2022/01/07 17:10:07.997 +08:00] [ERROR] [syncer.go:2588] ["error detected when executing SQL job"] [task=test] [unit="binlog replication"] [error="[code=10006:class=database:scope=downstream:level=high], Message: startLocation: [position: (dm-integration-test-2566-hjl2n-pw6g6-bin|000001.000001, 2116), gtid-set: 23ef696b-6f99-11ec-80f0-ae184bbcf630:1-9], endLocation: [position: (dm-integration-test-2566-hjl2n-pw6g6-bin|000001.000001, 2255), gtid-set: ], origin SQL: [alter table t1 drop column c1]: execute statement failed: ALTER TABLE `drop_column_with_index`.`t1` DROP COLUMN `c1`, RawCause: Error 8200: can't drop column c1 with tidb_enable_change_multi_schema is disable"]

Anything else we need to know

• Does this test exist for other branches as well?

• Has there been a high frequency of failure lately?