From 085d55521caecd387559c52a145e38bc35fc4c57 Mon Sep 17 00:00:00 2001 From: yikeke Date: Fri, 21 Aug 2020 14:08:18 +0800 Subject: [PATCH 1/6] align https://github.com/pingcap/docs-dm/pull/227/ --- en/TOC.md | 2 +- en/faq.md | 2 +- en/handle-failed-sql-statements.md | 783 ++++++++++++++++++ en/skip-or-replace-abnormal-sql-statements.md | 560 ------------- zh/handle-failed-sql-statements.md | 6 +- 5 files changed, 788 insertions(+), 565 deletions(-) create mode 100644 en/handle-failed-sql-statements.md delete mode 100644 en/skip-or-replace-abnormal-sql-statements.md diff --git a/en/TOC.md b/en/TOC.md index 78e32f41f..892306a78 100644 --- a/en/TOC.md +++ b/en/TOC.md @@ -41,7 +41,7 @@ - [Pause a Task](pause-task.md) - [Resume a Task](resume-task.md) - [Stop a Task](stop-task.md) - - [Skip or Replace Abnormal SQL Statements](skip-or-replace-abnormal-sql-statements.md) + - [Handle Failed SQL Statements](handle-failed-sql-statements.md) - [Manually Handle Sharding DDL Locks](feature-manually-handling-sharding-ddl-locks.md) - [Manage Table Schema during Migration](manage-schema.md) - [Handle Alerts](handle-alerts.md) diff --git a/en/faq.md b/en/faq.md index 572f8862c..bfcd27795 100644 --- a/en/faq.md +++ b/en/faq.md @@ -22,7 +22,7 @@ DM will attempt to split a single statement containing multiple DDL change opera ## How to handle incompatible DDL statements? -When you encounter a DDL statement unsupported by TiDB, you need to manually handle it using dmctl (skipping the DDL statement or replacing the DDL statement with a specified DDL statement). For details, see [Skip or replace abnormal SQL statements](skip-or-replace-abnormal-sql-statements.md). +When you encounter a DDL statement unsupported by TiDB, you need to manually handle it using dmctl (skipping the DDL statement or replacing the DDL statement with a specified DDL statement). For details, see [Handle failed SQL statements](handle-failed-sql-statements.md). > **Note:** > diff --git a/en/handle-failed-sql-statements.md b/en/handle-failed-sql-statements.md new file mode 100644 index 000000000..2782697bf --- /dev/null +++ b/en/handle-failed-sql-statements.md @@ -0,0 +1,783 @@ +--- +title: Handle Failed SQL Statements +summary: Learn how to handle failed SQL statements when you're using the TiDB Data Migration tool to migrate data. +aliases: ['/docs/tidb-data-migration/dev/skip-or-replace-abnormal-sql-statements/','/tidb-data-migration/dev/skip-or-replace-abnormal-sql-statements'] +--- + +# Handle Failed SQL Statements + +This document introduces how to handle failed SQL statements when you're using the TiDB Data Migration tool to migrate data.. + +Currently, TiDB is not completely compatible with all MySQL syntax (see [the DDL statements supported by TiDB](https://pingcap.com/docs/dev/reference/mysql-compatibility/#ddl)). Therefore, when DM is replicating data from MySQL to TiDB and TiDB does not support the corresponding SQL statement, an error might occur and break the replication process. In this case, you can use the `handle-error` command of DM to resume the replication. + +## Restrictions + +If it is unacceptable in the actual production environment that the failed DDL statement is skipped in the downstream TiDB and it cannot be replaced with other DDL statements, then do not use this command. + +For example, `DROP PRIMARY KEY`. In this scenario, you can only create a new table in the downstream with the new table schema (after executing the DDL statement), and re-import all the data into this new table. + +## Supported scenarios + +During the replication, the DDL statement unsupported by TiDB is executed in the upstream and replicated to the downstream, and as a result, the replication task gets interrupted. + +- If it is acceptable that this DDL statement is skipped in the downstream TiDB, then you can use `handle-error skip` to skip replicating this DDL statement and resume the replication. +- If it is acceptable that this DDL statement is replaced with other DDL statements, then you can use `handle-error replace` to replace this DDL statement and resume the replication. + +## Command + +When you use dmctl to manually handle the failed SQL statements, the commonly used commands include `query-status` and `handle-error`. + +### query-status + +The `query-status` command is used to query the current status of items such as the subtask and the relay unit in each MySQL instance. For details, see [query status](query-status.md). + +### handle-error + +The `handle-error` command is used to handle the failed SQL statements. + +### 命令用法 + +```bash +» handle-error -h +``` + +``` +Usage: + dmctl handle-error [-s source ...] [-b binlog-pos] [replace-sql1;replace-sql2;] [flags] + +Flags: + -b, --binlog-pos string position used to match binlog event if matched the handler-error operation will be applied. The format like "mysql-bin|000001.000003:3270" + -h, --help help for handle-error + +Global Flags: + -s, --source strings MySQL Source ID +``` + +### Flags descriptions + ++ `task-name`: + - Non-flag parameter, string, required + - `task-name` specifies the name of the task in which the presetted operation is going to be executed. + ++ `source`: + - Flag parameter, string, `--source` + - `source` specifies the MySQL instance in which the preset operation is to be executed. + ++ `skip`: Skip the error + ++ `replace`: Replace the failed SQL statement + ++ `revert`: Reset the previous skip/replace operation before the error occurs (only reset it when the previous skip/replace operation has not finally taken effect) + ++ `binlog-pos`: + - Flag parameter, string, `--binlog-pos` + - If it is not specified, DM automatically handles the currently failed SQL statement + - If it is specified, the skip operation is executed when `binlog-pos` matches with the position of the binlog event. The format is `binlog-filename:binlog-pos`, for example, `mysql-bin|000001.000003:3270`. + - When the replication returns an error, the binlog position can be obtained from `position` in `startLocation` returned by `query-status`. + +## Usage examples + +### Skip SQL if the replication gets interrupted + +#### Non-shard-merge scenario + +Assume that you need to replicate the upstream table `db1.tbl1` to the downstream TiDB. The initial table schema is: + +{{< copyable "sql" >}} + +```sql +SHOW CREATE TABLE db1.tbl1; +``` + +```sql ++-------+--------------------------------------------------+ +| Table | Create Table | ++-------+--------------------------------------------------+ +| tbl1 | CREATE TABLE `tbl1` ( + `c1` int(11) NOT NULL, + `c2` decimal(11,3) DEFAULT NULL, + PRIMARY KEY (`c1`) +) ENGINE=InnoDB DEFAULT CHARSET=latin1 | ++-------+--------------------------------------------------+ +``` + +Now, the following DDL statement is executed in the upstream to alter the table schema (namely, alter DECIMAL(11, 3) of c2 into DECIMAL(10, 3)): + +{{< copyable "sql" >}} + +```sql +ALTER TABLE db1.tbl1 CHANGE c2 c2 DECIMAL (10, 3); +``` + +Because this DDL statement is not supported by TiDB, the replication task of DM gets interrupted. Execute the `query-status ` command, and you can see the following error: + +``` +ERROR 8200 (HY000): Unsupported modify column: can't change decimal column precision +``` + +Assume that it is acceptable in the actual production environment that this DDL statement is not executed in the downstream TiDB (namely, the original table schema is retained). Then you can use `handle-error skip` to skip this DDL statement to resume the replication. The procedures are as follows: + +1. Execute `handle-error skip` to skip the currently failed DDL statement: + + {{< copyable "" >}} + + ```bash + » handle-error test skip + ``` + + ``` + { + "result": true, + "msg": "", + "sources": [ + { + "result": true, + "msg": "", + "source": "mysql-replica-01", + "worker": "worker1" + } + ] + } + ``` + +2. Execute `query-status ` to view the task status: + + {{< copyable "" >}} + + ```bash + » query-status test + ``` + +
See the execution result. + + ``` + { + "result": true, + "msg": "", + "sources": [ + { + "result": true, + "msg": "", + "sourceStatus": { + "source": "mysql-replica-01", + "worker": "worker1", + "result": null, + "relayStatus": null + }, + "subTaskStatus": [ + { + "name": "test", + "stage": "Running", + "unit": "Sync", + "result": null, + "unresolvedDDLLockID": "", + "sync": { + "totalEvents": "4", + "totalTps": "0", + "recentTps": "0", + "masterBinlog": "(DESKTOP-T561TSO-bin.000001, 2388)", + "masterBinlogGtid": "143bdef3-dd4a-11ea-8b00-00155de45f57:1-10", + "syncerBinlog": "(DESKTOP-T561TSO-bin.000001, 2388)", + "syncerBinlogGtid": "143bdef3-dd4a-11ea-8b00-00155de45f57:1-4", + "blockingDDLs": [ + ], + "unresolvedGroups": [ + ], + "synced": true, + "binlogType": "remote" + } + } + ] + } + ] + } + ``` + +
+ + You can see that the task runs normally and the wrong DDL is skipped. + +#### Shard merge scenario + +Assume that you need to merge and replicate the following four tables in the upstream to one same table ``` `shard_db`.`shard_table` ``` in the downstream. The task mode is "pessimistic". + +The initial table schema is: + +{{< copyable "sql" >}} + +```sql +SHOW CREATE TABLE shard_db.shard_table; +``` + +```sql ++-------+-----------------------------------------------------------------------------------------------------------+ +| Table | Create Table | ++-------+-----------------------------------------------------------------------------------------------------------+ +| tb | CREATE TABLE `shard_table` ( + `id` int(11) DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_bin | ++-------+-----------------------------------------------------------------------------------------------------------+ +``` + +Now, execute the following DDL statement to all upstream sharded tables to alter their character set: + +{{< copyable "sql" >}} + +```sql +ALTER TABLE `shard_db_*`.`shard_table_*` CHARACTER SET LATIN1 COLLATE LATIN1_DANISH_CI; +``` + +Because this DDL statement is not supported by TiDB, the replication task of DM gets interrupted. Execute the `query-status` command, and you can see the following errors reported by the `shard_db_1`.`shard_table_1` table in MySQL instance 1 and the `shard_db_2`.`shard_table_1` table in MySQL instance 2: + +``` +{ + "Message": "cannot track DDL: ALTER TABLE `shard_db_1`.`shard_table_1` CHARACTER SET UTF8 COLLATE UTF8_UNICODE_CI", + "RawCause": "[ddl:8200]Unsupported modify charset from latin1 to utf8" +} +``` + +``` +{ + "Message": "cannot track DDL: ALTER TABLE `shard_db_2`.`shard_table_1` CHARACTER SET UTF8 COLLATE UTF8_UNICODE_CI", + "RawCause": "[ddl:8200]Unsupported modify charset from latin1 to utf8" +} +``` + +Assume that it is acceptable in the actual production environment that this DDL statement is not executed in the downstream TiDB (namely, the original table schema is retained). Then you can use `handle-error skip` to skip this DDL statement to resume the replication. The procedures are as follows: + +1. Execute `handle-error skip` to skip the currently failed DDL statements in MySQL instance 1 and 2: + + {{< copyable "" >}} + + ```bash + » handle-error test skip + ``` + + ``` + { + "result": true, + "msg": "", + "sources": [ + { + "result": true, + "msg": "", + "source": "mysql-replica-01", + "worker": "worker1" + }, + { + "result": true, + "msg": "", + "source": "mysql-replica-02", + "worker": "worker2" + } + ] + } + ``` + +2. Execute the `query-status` command, and you can see the errors reported by the `shard_db_1`.`shard_table_1` table in MySQL instance 1 and the `shard_db_2`.`shard_table_1` table in MySQL instance 2: + + ``` + { + "Message": "cannot track DDL: ALTER TABLE `shard_db_1`.`shard_table_2` CHARACTER SET UTF8 COLLATE UTF8_UNICODE_CI", + "RawCause": "[ddl:8200]Unsupported modify charset from latin1 to utf8" + } + ``` + + ``` + { + "Message": "cannot track DDL: ALTER TABLE `shard_db_2`.`shard_table_2` CHARACTER SET UTF8 COLLATE UTF8_UNICODE_CI", + "RawCause": "[ddl:8200]Unsupported modify charset from latin1 to utf8" + } + ``` + +3. Execute `handle-error skip` again to skip the currently failed DDL statements in MySQL instance 1 and 2: + + {{< copyable "" >}} + + ```bash + » handle-error test skip + ``` + + ``` + { + "result": true, + "msg": "", + "sources": [ + { + "result": true, + "msg": "", + "source": "mysql-replica-01", + "worker": "worker1" + }, + { + "result": true, + "msg": "", + "source": "mysql-replica-02", + "worker": "worker2" + } + ] + } + ``` + +4. Use `query-status ` to view the task status: + + {{< copyable "" >}} + + ```bash + » query-status test + ``` + +
See the execution result. + + ``` + { + "result": true, + "msg": "", + "sources": [ + { + "result": true, + "msg": "", + "sourceStatus": { + "source": "mysql-replica-01", + "worker": "worker1", + "result": null, + "relayStatus": null + }, + "subTaskStatus": [ + { + "name": "test", + "stage": "Running", + "unit": "Sync", + "result": null, + "unresolvedDDLLockID": "", + "sync": { + "totalEvents": "4", + "totalTps": "0", + "recentTps": "0", + "masterBinlog": "(DESKTOP-T561TSO-bin.000001, 2388)", + "masterBinlogGtid": "143bdef3-dd4a-11ea-8b00-00155de45f57:1-10", + "syncerBinlog": "(DESKTOP-T561TSO-bin.000001, 2388)", + "syncerBinlogGtid": "143bdef3-dd4a-11ea-8b00-00155de45f57:1-4", + "blockingDDLs": [ + ], + "unresolvedGroups": [ + ], + "synced": true, + "binlogType": "remote" + } + } + ] + }, + { + "result": true, + "msg": "", + "sourceStatus": { + "source": "mysql-replica-02", + "worker": "worker2", + "result": null, + "relayStatus": null + }, + "subTaskStatus": [ + { + "name": "test", + "stage": "Running", + "unit": "Sync", + "result": null, + "unresolvedDDLLockID": "", + "sync": { + "totalEvents": "4", + "totalTps": "0", + "recentTps": "0", + "masterBinlog": "(DESKTOP-T561TSO-bin.000001, 2388)", + "masterBinlogGtid": "143bdef3-dd4a-11ea-8b00-00155de45f57:1-10", + "syncerBinlog": "(DESKTOP-T561TSO-bin.000001, 2388)", + "syncerBinlogGtid": "143bdef3-dd4a-11ea-8b00-00155de45f57:1-4", + "blockingDDLs": [ + ], + "unresolvedGroups": [ + ], + "synced": true, + "binlogType": "remote" + } + } + ] + } + ] + } + ``` + +
+ + You can see that the task runs normally with no error and all four wrong DDL statements are skipped. + +### Replace SQL if the replication gets interrupted + +#### Non-shard-merge scenario + +Assume that you need to replicate the upstream table `db1.tbl1` to the downstream TiDB. The initial table schema is: + +{{< copyable "sql" >}} + +```sql +SHOW CREATE TABLE db1.tbl1; +``` + +```SQL ++-------+-----------------------------------------------------------------------------------------------------------+ +| Table | Create Table | ++-------+-----------------------------------------------------------------------------------------------------------+ +| tb | CREATE TABLE `tbl1` ( + `id` int(11) DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_bin | ++-------+-----------------------------------------------------------------------------------------------------------+ +``` + +Now, perform the following DDL operation in the upstream to add a new column with the UNIQUE constraint: + +{{< copyable "sql" >}} + +```sql +ALTER TABLE `db1`.`tbl1` ADD COLUMN new_col INT UNIQUE; +``` + +Because this DDL statement is not supported by TiDB, the replication task gets interrupted. Execute the `query-status` command, and you can see the following error: + +``` +{ + "Message": "cannot track DDL: ALTER TABLE `db1`.`tbl1` ADD COLUMN `new_col` INT UNIQUE KEY", + "RawCause": "[ddl:8200]unsupported add column 'new_col' constraint UNIQUE KEY when altering 'db1.tbl1'", +} +``` + +You can replace this DDL statement with two equivalent DDL statements. The steps are as follows: + +1. Replace the wrong DDL statement by the following command: + + {{< copyable "" >}} + + ```bash + » handle-error test replace "ALTER TABLE `db1`.`tbl1` ADD COLUMN `new_col` INT;ALTER TABLE `db1`.`tbl1` ADD UNIQUE(`new_col`)"; + ``` + + ``` + { + "result": true, + "msg": "", + "sources": [ + { + "result": true, + "msg": "", + "source": "mysql-replica-01", + "worker": "worker1" + } + ] + } + ``` + +2. Use `query-status ` to view the task status: + + {{< copyable "" >}} + + ```bash + » query-status test + ``` + +
See the execution result. + + ``` + { + "result": true, + "msg": "", + "sources": [ + { + "result": true, + "msg": "", + "sourceStatus": { + "source": "mysql-replica-01", + "worker": "worker1", + "result": null, + "relayStatus": null + }, + "subTaskStatus": [ + { + "name": "test", + "stage": "Running", + "unit": "Sync", + "result": null, + "unresolvedDDLLockID": "", + "sync": { + "totalEvents": "4", + "totalTps": "0", + "recentTps": "0", + "masterBinlog": "(DESKTOP-T561TSO-bin.000001, 2388)", + "masterBinlogGtid": "143bdef3-dd4a-11ea-8b00-00155de45f57:1-10", + "syncerBinlog": "(DESKTOP-T561TSO-bin.000001, 2388)", + "syncerBinlogGtid": "143bdef3-dd4a-11ea-8b00-00155de45f57:1-4", + "blockingDDLs": [ + ], + "unresolvedGroups": [ + ], + "synced": true, + "binlogType": "remote" + } + } + ] + } + ] + } + ``` + +
+ + You can see that the task runs normally and the wrong DDL statement is replaced by new DDL statements that execute successfully. + +#### Shard merge scenario + +Assume that you need to merge and replicate the following four tables in the upstream to one same table ``` `shard_db`.`shard_table` ``` in the downstream. The task mode is "pessimistic". + +- In the MySQL instance 1, there is a schema `shard_db_1`, which has two tables `shard_table_1` and `shard_table_2`. +- In the MySQL instance 2, there is a schema `shard_db_2`, which has two tables `shard_table_1` and `shard_table_2`. + +The initial table schema is: + +{{< copyable "sql" >}} + +```sql +SHOW CREATE TABLE shard_db.shard_table; +``` + +```sql ++-------+-----------------------------------------------------------------------------------------------------------+ +| Table | Create Table | ++-------+-----------------------------------------------------------------------------------------------------------+ +| tb | CREATE TABLE `shard_table` ( + `id` int(11) DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_bin | ++-------+-----------------------------------------------------------------------------------------------------------+ +``` + +Now, perform the following DDL operation to all upstream sharded tables to add a new column with the UNIQUE constraint: + +{{< copyable "sql" >}} + +```sql +ALTER TABLE `shard_db_*`.`shard_table_*` ADD COLUMN new_col INT UNIQUE; +``` + +Because this DDL statement is not supported by TiDB, the replication task gets interrupted. Execute the `query-status` command, and you can see the following errors reported by the `shard_db_1`.`shard_table_1` table in MySQL instance 1 and the `shard_db_2`.`shard_table_1` table in MySQL instance 2: + +``` +{ + "Message": "cannot track DDL: ALTER TABLE `shard_db_1`.`shard_table_1` ADD COLUMN `new_col` INT UNIQUE KEY", + "RawCause": "[ddl:8200]unsupported add column 'new_col' constraint UNIQUE KEY when altering 'shard_db_1.shard_table_1'", +} +``` + +``` +{ + "Message": "cannot track DDL: ALTER TABLE `shard_db_2`.`shard_table_1` ADD COLUMN `new_col` INT UNIQUE KEY", + "RawCause": "[ddl:8200]unsupported add column 'new_col' constraint UNIQUE KEY when altering 'shard_db_2.shard_table_1'", +} +``` + +You can replace this DDL statement with two equivalent DDL statements. The steps are as follows: + +1. Replace the wrong DDL statements respectively in MySQL instance 1 and MySQL instance 2 by the following commands: + + {{< copyable "" >}} + + ```bash + » handle-error test -s mysql-replica-01 replace "ALTER TABLE `shard_db_1`.`shard_table_1` ADD COLUMN `new_col` INT;ALTER TABLE `shard_db_1`.`shard_table_1` ADD UNIQUE(`new_col`)"; + ``` + + ``` + { + "result": true, + "msg": "", + "sources": [ + { + "result": true, + "msg": "", + "source": "mysql-replica-01", + "worker": "worker1" + } + ] + } + ``` + + {{< copyable "" >}} + + ```bash + » handle-error test -s mysql-replica-02 replace "ALTER TABLE `shard_db_2`.`shard_table_1` ADD COLUMN `new_col` INT;ALTER TABLE `shard_db_2`.`shard_table_1` ADD UNIQUE(`new_col`)"; + ``` + + ``` + { + "result": true, + "msg": "", + "sources": [ + { + "result": true, + "msg": "", + "source": "mysql-replica-02", + "worker": "worker2" + } + ] + } + ``` + +2. Use `query-status ` to view the task status, and you can see the following errors reported by the `shard_db_1`.`shard_table_1` table in MySQL instance 1 and the `shard_db_2`.`shard_table_1` table in MySQL instance 2: + + ``` + { + "Message": "detect inconsistent DDL sequence from source ... ddls: [ALTER TABLE `shard_db`.`tb` ADD COLUMN `new_col` INT UNIQUE KEY] source: `shard_db_1`.`shard_table_2`], right DDL sequence should be ..." + } + ``` + + ``` + { + "Message": "detect inconsistent DDL sequence from source ... ddls: [ALTER TABLE `shard_db`.`tb` ADD COLUMN `new_col` INT UNIQUE KEY] source: `shard_db_2`.`shard_table_2`], right DDL sequence should be ..." + } + ``` + +3. Execute `handle-error replace` again to replace the wrong DDL statements in MySQL instance 1 and 2: + + {{< copyable "" >}} + + ```bash + » handle-error test -s mysql-replica-01 replace "ALTER TABLE `shard_db_1`.`shard_table_2` ADD COLUMN `new_col` INT;ALTER TABLE `shard_db_1`.`shard_table_2` ADD UNIQUE(`new_col`)"; + ``` + + ``` + { + "result": true, + "msg": "", + "sources": [ + { + "result": true, + "msg": "", + "source": "mysql-replica-01", + "worker": "worker1" + } + ] + } + ``` + + {{< copyable "" >}} + + ```bash + » handle-error test -s mysql-replica-02 replace "ALTER TABLE `shard_db_2`.`shard_table_2` ADD COLUMN `new_col` INT;ALTER TABLE `shard_db_2`.`shard_table_2` ADD UNIQUE(`new_col`)"; + ``` + + ``` + { + "result": true, + "msg": "", + "sources": [ + { + "result": true, + "msg": "", + "source": "mysql-replica-02", + "worker": "worker2" + } + ] + } + ``` + +4. Use `query-status ` to view the task status: + + {{< copyable "" >}} + + ```bash + » query-status test + ``` + +
See the execution result. + + ``` + { + "result": true, + "msg": "", + "sources": [ + { + "result": true, + "msg": "", + "sourceStatus": { + "source": "mysql-replica-01", + "worker": "worker1", + "result": null, + "relayStatus": null + }, + "subTaskStatus": [ + { + "name": "test", + "stage": "Running", + "unit": "Sync", + "result": null, + "unresolvedDDLLockID": "", + "sync": { + "totalEvents": "4", + "totalTps": "0", + "recentTps": "0", + "masterBinlog": "(DESKTOP-T561TSO-bin.000001, 2388)", + "masterBinlogGtid": "143bdef3-dd4a-11ea-8b00-00155de45f57:1-10", + "syncerBinlog": "(DESKTOP-T561TSO-bin.000001, 2388)", + "syncerBinlogGtid": "143bdef3-dd4a-11ea-8b00-00155de45f57:1-4", + "blockingDDLs": [ + ], + "unresolvedGroups": [ + ], + "unresolvedGroups": [ + ], + "synced": true, + "binlogType": "remote" + } + } + ] + }, + { + "result": true, + "msg": "", + "sourceStatus": { + "source": "mysql-replica-02", + "worker": "worker2", + "result": null, + "relayStatus": null + }, + "subTaskStatus": [ + { + "name": "test", + "stage": "Running", + "unit": "Sync", + "result": null, + "unresolvedDDLLockID": "", + "sync": { + "totalEvents": "4", + "totalTps": "0", + "recentTps": "0", + "masterBinlog": "(DESKTOP-T561TSO-bin.000001, 2388)", + "masterBinlogGtid": "143bdef3-dd4a-11ea-8b00-00155de45f57:1-10", + "syncerBinlog": "(DESKTOP-T561TSO-bin.000001, 2388)", + "syncerBinlogGtid": "143bdef3-dd4a-11ea-8b00-00155de45f57:1-4", + "blockingDDLs": [ + ], + "unresolvedGroups": [ + ], + "unresolvedGroups": [ + ], + "synced": try, + "binlogType": "remote" + } + } + ] + } + ] + } + ``` + +
+ + You can see that the task runs normally with no error and all four wrong DDL statements are replaced. diff --git a/en/skip-or-replace-abnormal-sql-statements.md b/en/skip-or-replace-abnormal-sql-statements.md deleted file mode 100644 index 445bd966d..000000000 --- a/en/skip-or-replace-abnormal-sql-statements.md +++ /dev/null @@ -1,560 +0,0 @@ ---- -title: Skip or Replace Abnormal SQL Statements -summary: Learn how to skip or replace abnormal SQL statements when you use DM. -aliases: ['/docs/tidb-data-migration/dev/skip-or-replace-abnormal-sql-statements/'] ---- - -# Skip or Replace Abnormal SQL Statements - -This document introduces how to handle abnormal SQL statements using DM. - -> **Note:** -> -> The `sql-skip` and `sql-replace` commands are currently not supported. - -Currently, TiDB is not completely compatible with all MySQL syntax (see [the DDL statements supported by TiDB](https://pingcap.com/docs/dev/reference/mysql-compatibility/#ddl)). Therefore, when DM is replicating data from MySQL to TiDB and TiDB does not support the corresponding SQL statement, an error might occur and break the replication process. In this case, there are two ways to resume the replication: - -- Use dmctl to manually skip the binlog event to which this SQL statement corresponds - -- Use dmctl to manually replace the corresponding binlog event with other specified SQL statements that should be executed to the downstream later - -If you know in advance that an unsupported SQL statement is going to be replicated, you can also use dmctl to manually preset the skip or replace operation, which is automatically executed when DM replicates the corresponding binlog event into the downstream and thus avoid breaking the replication. - -## Restrictions - -- The skip or replace operation is a one-time operation that is only used to skip or replace the SQL statement unsupported by the downstream TiDB. Do not handle other replication errors with this approach. - - For other replication errors, try to handle them using [Block and allow table lists](key-features.md#block-and-allow-table-lists) or [Binlog event filtering](key-features.md#binlog-event-filter). - -- If it is unacceptable in the actual production environment that the abnormal DDL statement is skipped in the downstream TiDB and it cannot be replaced with other DDL statements, then do not use this approach. - - For example: `DROP PRIMARY KEY` - - In this scenario, you can only create a new table in the downstream with the new table schema (after executing the DDL statement), and re-import all the data into this new table. - -- A single skip or replace operation targets at a single binlog event. - -- `--sharding` is only used to preset the operation to the sharding group. You must preset it before executing the DDL statement and presetting it after executing the DDL is not allowed. - - `--sharding` only supports presetting operations, and in this mode, you can only use `--sql-pattern` to match the binlog event. - - For the principles of replicating sharding DDL statements using DM, see [Merge and replicate data from sharded tables](feature-shard-merge.md#principles) - -## Match the binlog event - -When the replication task gets interrupted because of the SQL execution error, you can obtain the position of the corresponding binlog event by using `query-error`. When you execute `sql-skip` or `sql-replace`, you can specify the position to match the binlog event. - -However, when you try to avoid breaking the replication by actively handling unsupported SQL statements, you cannot know in advance the position of the binlog event, so you need another approach to match the subsequent binlog events. - -In DM, two modes of matching the binlog event are supported (you can only choose one mode from below): - -1. binlog position: the position information of the binlog event - - - The binlog position is given by `--binlog-pos` in the command, and the format is `binlog-filename:binlog-pos`, for example, `mysql-bin|000001.000003:3270`. - - The format of the binlog filename in DM is not completely consistent with that in the upstream MySQL. - - When the replication error occurs, the position can be directly obtained from `failedBinlogPosition` returned by `query-error`. - -2. DDL pattern: the regular expression (only for the DDL statement) matching mode - - - The DDL pattern is given by `--sql-pattern` in the command, for example, to match ```ALTER TABLE `db2`.`tbl2` DROP COLUMN `c2` ```, the corresponding regular expression should be ```~(?i)ALTER\s+TABLE\s+`db2`.`tbl2`\s+DROP\s+COLUMN\s+`c2` ```. - - The regular expression must be prefixed with `~` and cannot contain any common space (you can replace the space with `\s` or `\s+` in the string). - -In the scenario of merging and replicating data from sharded tables, if you need DM to automatically select a DDL lock owner to execute the skip or replace operation, then you must use the DDL pattern matching mode because the binlog positions corresponding to the DDL statements on different DM-workers have no logical connection and are hard to confirm. - -> **Note:** -> -> - You can only register one operator (specified by `--binlog-pos`) for one binlog event. The previous one can be overwritten by the newly registered operator. -> - Do not specify an operator for one binlog event by using `--binlog-pos` and `--sql-pattern` at the same time. -> - The operator is deleted once it successfully matches the binlog event (not after the execution succeeds). If you need to match again (using `--sql-pattern`) later, you have to register a new operator. - -## Supported scenarios - -- Scenario 1: during the replication, the DDL statement unsupported by TiDB is executed in the upstream and replicated to the downstream, and as a result, the replication task gets interrupted. - - - If it is acceptable that this DDL statement is skipped in the downstream TiDB, then you can use `sql-skip` to resume the replication. - - If it is acceptable that this DDL statement is replaced with other DDL statements, then you can use `sql-replace` to resume the replication. - -- Scenario 2: during the replication, you know in advance that an unsupported SQL statement is going to be replicated, so you can handle it beforehand to avoid breaking the replication. - - - If it is acceptable that this DDL statement is skipped in the downstream TiDB, then you can use `sql-skip` to preset an operation to automatically skip this DDL statement when it needs to be executed. - - If it is acceptable that this DDL statement is replaced with other DDL statements, then you can use `sql-replace` to preset an operation to automatically replace this DDL statement when it needs to be executed. - -## Implementation principles - -In DM, simplified procedures of incremental data replication can be described as follows: - -1. The relay unit is used as a secondary database of the upstream MySQL to fetch the binlog that is persisted in the local storage as the relay log. - -2. The binlog replication unit (sync) reads the local relay log to obtain the binlog event. - -3. The binlog replication unit parses the binlog event and builds the DDL/DML statements, and then replicates these statements to the downstream TiDB. - -When the binlog replication unit is parsing the binlog event and replicating data to the downstream, the replication process might get interrupted because the corresponding SQL statement is not supported by TiDB. - -In DM, you can register some skip or replace operators for the binlog event. Before replicating the SQL statements to the downstream, DM compares the current binlog event information(position, DDL statement) with registered operators. If the position or the DDL matches with a registered operator, it executes the operation corresponding to the operator and then remove this operator. - -**Use `sql-skip` / `sql-replace` to resume the replication** - -1. Use `sql-skip` or `sql-replace` to register an operator for the specified binlog position or DDL pattern. - -2. Use `resume-task` to resume the replication task. - -3. Regain and re-parse the binlog event that causes the replication error. - -4. The binlog event successfully matches with the registered operator in step 1. - -5. Execute the skip or replace operation corresponding to the operator and then the replication task continues. - -**Use `sql-skip` / `sql-replace` to preset operations to avoid breaking the replication** - -1. Use `sql-skip` or `sql-replace` to register an operator for the specified DDL pattern. - -2. Parse the relay log to obtain the binlog event. - -3. The binlog event (including the SQL statements unsupported by TiDB) successfully matches with the registered operator in step 1. - -4. Execute the skip or replace operation corresponding to the operator and then the replication task continues and does not get interrupted. - -**Use `sql-skip` / `sql-replace` to preset operations to avoid breaking the replication in the scenario of merging and replicating data from sharded tables** - -1. Use `sql-skip` or `sql-replace` to register an operator (on DM-master) for the specified DDL pattern. - -2. Each DM-worker parses the relay log to obtain the binlog event. - -3. DM-master coordinates the DDL lock replication among DM-workers. - -4. DM-master checks if the DDL lock replication succeeds, and sends the registered operator in step 1 to the DDL lock owner. - -5. DM-master requests the DDL lock owner to execute the DDL statement. - -6. The DDL statement that is to be executed by the DDL lock owner successfully matches with the received operator in step 4. - -7. Execute the skip or replace operation corresponding to the operator and then the replication task continues. - -## Command - -When you use dmctl to manually handle the SQL statements unsupported by TiDB, the commonly used commands include `query-status`, `query-error`, `sql-skip` and `sql-replace`. - -### query-status - -`query-status` allows you to query the current status of items such as the subtask and the relay unit in each DM-worker. For details, see [query status](query-status.md). - -### query-error - -`query-error` allows you to query errors that occur during the running of MySQL instance subtasks. - -### sql-skip - -`sql-skip` allows you to preset a skip operation that is to be executed when the position or the SQL statement of the binlog event matches with the specified `binlog-pos` or `sql-pattern`. - -#### Command usage - -```bash -sql-skip <--source mysql-replica-01> [--binlog-pos=mysql-bin|000001.000003:3270] [--sql-pattern=~(?i)ALTER\s+TABLE\s+`db1`.`tbl1`\s+ADD\s+COLUMN\s+col1\s+INT] [--sharding] -``` - -#### Flags description - -+ `source`: - - Flag parameter, string, `--source` - - If `--sharding` is not specified, `source` is required; if `--sharding` is specified, `source` is forbidden to use. - - `source` specifies the MySQL instance in which the preset operation is to be executed. - -+ `binlog-pos`: - - Flag parameter, string, `--binlog-pos` - - You must specify `binlog-pos` or `--sql-pattern`, and you must not specify both. - - If it is specified, the skip operation is executed when `binlog-pos` matches with the position of the binlog event. The format is `binlog-filename:binlog-pos`, for example, `mysql-bin|000001.000003:3270`. - - When the replication error occurs, the position can be obtained from `failedBinlogPosition` returned by `query-error`. - -+ `sql-pattern`: - - Flag parameter, string, `--sql-pattern` - - You must specify `--sql-pattern` or `binlog-pos`, and you must not specify both. - - If it is specified, the skip operation is executed when `sql-pattern` matches with the DDL statement (converted by the optional router-rule) of the binlog event. The format is a regular expression prefixed with `~`, for example, ```~(?i)ALTER\s+TABLE\s+`db1`.`tbl1`\s+ADD\s+COLUMN\s+col1\s+INT```. - - Common spaces are not supported in the regular expression temporarily. You can replace the space with `\s` or `\s+` if it is needed. - - The regular expression must be prefixed with `~`. For details, see [regular expression syntax](https://golang.org/pkg/regexp/syntax/#hdr-syntax). - - The schema/table name in the regular expression must be converted by the optional router-rule, so the converted name is consistent with the target schema/table name in the downstream. For example, if there are ``` `shard_db_1`.`shard_tbl_1` ``` in the upstream and ``` `shard_db`.`shard_tbl` ``` in the downstream, then you should match ``` `shard_db`.`shard_tbl` ```. - - The schema/table/column name in the regular expression should be marked by ``` ` ```, for example, ``` `db1`.`tbl1` ```. - -+ `sharding`: - - Flag parameter, boolean, `--sharding` - - If `--worker` is not specified, `sharding` is required; if `--worker` is specified, `sharding` is forbidden to use. - - If `sharding` is specified, it indicates that the presetted operation is going to be executed in the DDL lock owner during the sharding DDL replication. - -+ `task-name`: - - Non-flag parameter, string, required - - `task-name` specifies the name of the task in which the presetted operation is going to be executed. - -### sql-replace - -`sql-replace` allows you to preset a replace operation that is to be executed when the position or the SQL statement of the binlog event matches with the specified `binlog-pos` or `sql-pattern`. - -#### Command usage - -```bash -sql-replace <--source mysql-replica-01> [--binlog-pos=mysql-bin|000001.000003:3270] [--sql-pattern=~(?i)ALTER\s+TABLE\s+`db1`.`tbl1`\s+ADD\s+COLUMN\s+col1\s+INT] [--sharding] -``` - -#### Flags description - -+ `source`: - - same with `--source` of `sql-skip` - -+ `binlog-pos`: - - same with `--binlog-pos` of `sql-skip` - -+ `sql-pattern`: - - same with `--sql-pattern` of `sql-skip` - -+ `sharding`: - - same with `--sharding` of `sql-skip` - -+ `task-name`: - - same with `task-name` of `sql-skip` - -+ `SQLs`: - - Non-flag parameter, string, required - - `SQLs` specifies the new SQL statements that are going to replace the original binlog event. You should separate multiple SQL statements with `;`, for example, ```ALTER TABLE shard_db.shard_table drop index idx_c2;ALTER TABLE shard_db.shard_table DROP COLUMN c2;```. - -## Usage examples - -### Passively skip after the replication gets interrupted - -#### Application scenario - -Assume that you need to replicate the upstream table `db1.tbl1` to the downstream TiDB (not in the scenario of merging and replicating data from sharded tables). The initial table schema is: - -```sql -mysql> SHOW CREATE TABLE db1.tbl1; -+-------+--------------------------------------------------+ -| Table | Create Table | -+-------+--------------------------------------------------+ -| tbl1 | CREATE TABLE `tbl1` ( - `c1` int(11) NOT NULL, - `c2` decimal(11,3) DEFAULT NULL, - PRIMARY KEY (`c1`) -) ENGINE=InnoDB DEFAULT CHARSET=latin1 | -+-------+--------------------------------------------------+ -``` - -Now, the following DDL statement is executed in the upstream to alter the table schema (namely, alter DECIMAL(11, 3) of c2 into DECIMAL(10, 3)): - -```sql -ALTER TABLE db1.tbl1 CHANGE c2 c2 DECIMAL (10, 3); -``` - -Because this DDL statement is not supported by TiDB, the replication task of DM gets interrupted and reports the following error: - -```bash -exec sqls[[USE `db1`; ALTER TABLE `db1`.`tbl1` CHANGE COLUMN `c2` `c2` decimal(10,3);]] failed, -err:Error 1105: unsupported modify column length 10 is less than origin 11 -``` - -Now, if you query the status of the task using `query-status`, you can see that `stage` has changed into `Paused` and there is some related error description information in `errors`. - -To obtain the details about the error, you should use `query-error`. For example, you can execute `query-error test` to get the position of the failed binlog event (`failedBinlogPosition`), which is `mysql-bin|000001.000003:34642`. - -#### Passively skip the SQL statement - -Assume that it is acceptable in the actual production environment that this DDL statement is not executed in the downstream TiDB (namely, the original table schema is retained). Then you can use `sql-skip` to skip this DDL statement to resume the replication. The procedures are as follows: - -1. Use `query-error` to obtain the position of the failed binlog event. - - You can get the position from `failedBinlogPosition` returned by `query-error`. - - In this example, the position is `mysql-bin|000001.000003:34642`. - -2. Use `sql-skip` to preset a skip operation that is to be executed when DM replicates this binlog event to the downstream after using `resume-task`. - - ```bash - » sql-skip --source=mysql-replica-01 --binlog-pos=mysql-bin|000001.000003:34642 test - { - "result": true, - "msg": "", - "sources": [ - { - "result": true, - "msg": "", - "source": "", - "worker": "" - } - ] - } - ``` - - You can also view the following log in the DM-worker node corresponding to the source: - - ```bash - 2018/12/28 11:17:51 operator.go:121: [info] [sql-operator] set a new operator - uuid: 6bfcf30f-2841-4d70-9a34-28d7082bdbd7, pos: (mysql-bin|000001.000003, 34642), op: SKIP, args: - on replication unit - ``` - -3. Use `resume-task` to resume the replication task - - ```bash - » resume-task --source=mysql-replica-01 test - { - "op": "Resume", - "result": true, - "msg": "", - "sources": [ - { - "result": true, - "msg": "", - "source": "mysql-replica-01", - "worker": "worker1" - } - ] - } - ``` - - You can also view the following log in the DM-worker node corresponding to the source: - - ```bash - 2018/12/28 11:27:46 operator.go:158: [info] [sql-operator] binlog-pos (mysql-bin|000001.000003, 34642) matched, - applying operator uuid: 6bfcf30f-2841-4d70-9a34-28d7082bdbd7, pos: (mysql-bin|000001.000003, 34642), op: SKIP, args: - ``` - -4. Use `query-status` to guarantee that the `stage` of the task has changed into `Running`. - -5. Use `query-error` to guarantee that no DDL execution error exists. - -### Actively replace before the replication gets interrupted - -#### Application scenario - -Assume that you need to replicate the upstream table `db2.tbl2` to the downstream TiDB (not in the scenario of merging and replicating data from sharded tables). The initial table schema is: - -```sql -mysql> SHOW CREATE TABLE db2.tbl2; -+-------+--------------------------------------------------+ -| Table | Create Table | -+-------+--------------------------------------------------+ -| tbl2 | CREATE TABLE `tbl2` ( - `c1` int(11) NOT NULL, - `c2` int(11) DEFAULT NULL, - PRIMARY KEY (`c1`), - KEY `idx_c2` (`c2`) -) ENGINE=InnoDB DEFAULT CHARSET=latin1 | -+-------+--------------------------------------------------+ -``` - -Now, the following DDL statement is executed in the upstream to alter the table schema (namely, `DROP COLUMN c2`): - -```sql -ALTER TABLE db2.tbl2 DROP COLUMN c2; -``` - -Because this DDL statement is not supported by TiDB, the replication task of DM gets interrupted and reports the following error: - -```bash -exec sqls[[USE `db2`; ALTER TABLE `db2`.`tbl2` DROP COLUMN `c2`;]] failed, -err:Error 1105: can't drop column c2 with index covered now -``` - -**Assume that you know in advance that this DDL statement is not supported by TiDB before it is executed in the upstream.** Then you can use `sql-skip` or `sql-replace` to preset a skip or replace operation for this DDL statement. - -For this particular DDL statement, because dropping columns with the index is not temporarily supported by TiDB, you can use two new SQL statements to replace the original DDL, namely, DROP the index first and then DROP the column c2. - -#### Actively replace the SQL statement - -1. Design a matchable regular expression for the DDL statement (converted by the optional router-rule) to be executed in the upstream. - - The DDL statement to be executed in the upstream is `ALTER TABLE db2.tbl2 DROP COLUMN c2;`. - - Because its router-rule conversion does not exist, you can design the following regular expression: - - ```sql - ~(?i)ALTER\s+TABLE\s+`db2`.`tbl2`\s+DROP\s+COLUMN\s+`c2` - ``` - -2. Build new DDL statements that are used to replace this original DDL statement. - - ```sql - ALTER TABLE `db2`.`tbl2` DROP INDEX idx_c2;ALTER TABLE `db2`.`tbl2` DROP COLUMN `c2` - ``` - -3. Use `sql-replace` to preset a replace operation that is to be executed when DM replicates the corresponding binlog event to the downstream. - - ```bash - » sql-replace --source=mysql-replica-01 --sql-pattern=~(?i)ALTER\s+TABLE\s+`db2`.`tbl2`\s+DROP\s+COLUMN\s+`c2` test ALTER TABLE `db2`.`tbl2` DROP INDEX idx_c2;ALTER TABLE `db2`.`tbl2` DROP COLUMN `c2` - { - "result": true, - "msg": "", - "sources": [ - { - "result": true, - "msg": "", - "source": "", - "worker": "" - } - ] - } - ``` - - You can also view the following log in the DM-worker node corresponding to the source: - - ```bash - 2018/12/28 15:33:13 operator.go:121: [info] [sql-operator] set a new operator - uuid: c699a18a-8e75-47eb-8e7e-0e5abde2053c, pattern: ~(?i)ALTER\s+TABLE\s+`db2`.`tbl2`\s+DROP\s+COLUMN\s+`c2`, - op: REPLACE, args: ALTER TABLE `db2`.`tbl2` DROP INDEX idx_c2; ALTER TABLE `db2`.`tbl2` DROP COLUMN `c2` - on replication unit - ``` - -4. Execute the DDL statements in the upstream MySQL. - -5. Check if the downstream table schema is altered successfully, and you can view the following log in the corresponding DM-worker node: - - ```bash - 2018/12/28 15:33:45 operator.go:158: [info] [sql-operator] - sql-pattern ~(?i)ALTER\s+TABLE\s+`db2`.`tbl2`\s+DROP\s+COLUMN\s+`c2` matched SQL - USE `db2`; ALTER TABLE `db2`.`tbl2` DROP COLUMN `c2`;, - applying operator uuid: c699a18a-8e75-47eb-8e7e-0e5abde2053c, - pattern: ~(?i)ALTER\s+TABLE\s+`db2`.`tbl2`\s+DROP\s+COLUMN\s+`c2`, - op: REPLACE, args: ALTER TABLE `db2`.`tbl2` DROP INDEX idx_c2; ALTER TABLE `db2`.`tbl2` DROP COLUMN `c2` - ``` - -6. Use `query-status` to guarantee that the `stage` of the task has been sustained as `Running`. - -7. Use `query-error` to guarantee that no DDL execution error exists. - -### Passively skip after the replication gets interrupted in the scenario of merging and replicating data from sharded tables - -#### Application scenario - -Assume that you need to merge and replicate multiple tables in multiple upstream MySQL instances to one same table in the downstream TiDB through multiple DM-workers. And the DDL statement unsupported by TiDB is executed to the upstream sharded tables. - -After DM-master coordinates the DDL replication through the DDL lock and requests the DDL lock owner to execute the DDL statement to the downstream, the replication gets interrupted because this DDL statement is not supported by TiDB. - -#### Passively skip the SQL statement - -In the scenario of merging and replicating data from sharded tables, passively skipping the unsupported DDL statement has the similar steps with [Passively skip after the replication gets interrupted](#passively-skip-after-the-replication-gets-interrupted). - -There are two major differences between the two scenarios as follows. In the scenario of merging and replicating data from sharded tables: - -1. You just need the DDL lock owner to execute `sql-skip` (`--worker={DDL-lock-owner}`). - -2. You just need the DDL lock owner to execute `resume-task` (`--worker={DDL-lock-owner}`). - -### Actively replace before the replication gets interrupted in the scenario of merging and replicating data from sharded tables - -#### Application scenario - -Assume that you need to merge and replicate the following four tables in the upstream to one same table ``` `shard_db`.`shard_table` ``` in the downstream: - -- In the MySQL instance 1, there is a schema `shard_db_1`, which has two tables `shard_table_1` and `shard_table_2`. -- In the MySQL instance 2, there is a schema `shard_db_2`, which has two tables `shard_table_1` and `shard_table_2`. - -The initial table schema is: - -```sql -mysql> SHOW CREATE TABLE shard_db_1.shard_table_1; -+---------------+------------------------------------------+ -| Table | Create Table | -+---------------+------------------------------------------+ -| shard_table_1 | CREATE TABLE `shard_table_1` ( - `c1` int(11) NOT NULL, - `c2` int(11) DEFAULT NULL, - PRIMARY KEY (`c1`), - KEY `idx_c2` (`c2`) -) ENGINE=InnoDB DEFAULT CHARSET=latin1 | -+---------------+------------------------------------------+ -``` - -Now, the following DDL statement is executed to all upstream sharded tables to alter the table schemas (namely, `DROP COLUMN c2`): - -```sql -ALTER TABLE shard_db_*.shard_table_* DROP COLUMN c2; -``` - -When DM coordinates the two DM-workers to replicate this DDL statement through the sharding DDL lock and requests the DDL lock owner to execute the DDL statement to the downstream, because this DDL statement is not supported by TiDB, the replication task gets interrupted and report the following error: - -```bash -exec sqls[[USE `shard_db`; ALTER TABLE `shard_db`.`shard_table` DROP COLUMN `c2`;]] failed, -err:Error 1105: can't drop column c2 with index covered now -``` - -**Assume that you know in advance that this DDL statement is not supported by TiDB before it is executed in the upstream.** Then you can use `sql-skip` or `sql-replace` to preset a skip or replace operation for this DDL statement. - -For this particular DDL statement, because dropping columns with the index is not temporarily supported by TiDB, you can use two new SQL statements to replace the original DDL, namely, DROP the index first and then DROP the column c2. - -#### Actively replace the SQL statement - -1. Design a matchable regular expression for the DDL statement (converted by the optional router-rule) to be executed in the upstream. - - The DDL statement to be executed in the upstream is `ALTER TABLE shard_db_*.shard_table_* DROP COLUMN c2`. - - Because the table name should be converted into ``` `shard_db`.`shard_table` ``` by the router-rule, you can design the following regular expression: - - ```sql - ~(?i)ALTER\s+TABLE\s+`shard_db`.`shard_table`\s+DROP\s+COLUMN\s+`c2` - ``` - -2. Build new DDL statements that are used to replace this original DDL statement. - - ```sql - ALTER TABLE `shard_db`.`shard_table` DROP INDEX idx_c2;ALTER TABLE `shard_db`.`shard_table` DROP COLUMN `c2` - ``` - -3. Because this is in the scenario of merging and replicating data from sharded tables, you can use `--sharding` to automatically guarantee that the replace operation is only executed in the DDL lock owner. - -4. Use `sql-replace` to preset a replace operation that is to be executed when DM replicates the corresponding binlog event to the downstream. - - ```bash - » sql-replace --sharding --sql-pattern=~(?i)ALTER\s+TABLE\s+`shard_db`.`shard_table`\s+DROP\s+COLUMN\s+`c2` test ALTER TABLE `shard_db`.`shard_table` DROP INDEX idx_c2;ALTER TABLE `shard_db`.`shard_table` DROP COLUMN `c2` - { - "result": true, - "msg": "request with --sharding saved and will be sent to DDL lock's owner when resolving DDL lock", - "sources": [ - ] - } - ``` - - You can also view the following log in the **DM-master** node: - - ```bash - 2018/12/28 16:53:33 operator.go:105: [info] [sql-operator] set a new operator - uuid: eba35acd-6c5e-4bc3-b0b0-ae8bd1232351, request: name:"test" - op:REPLACE args:"ALTER TABLE `shard_db`.`shard_table` DROP INDEX idx_c2;" - args:"ALTER TABLE `shard_db`.`shard_table` DROP COLUMN `c2`" - sqlPattern:"~(?i)ALTER\\s+TABLE\\s+`shard_db`.`shard_table`\\s+DROP\\s+COLUMN\\s+`c2`" - sharding:true - ``` - -5. Execute the DDL statements to the sharded tables in the upstream MySQL instances. - -6. Check if the downstream table schema is altered successfully, and you can also view the following log in the DDL lock **owner** node: - - ```bash - 2018/12/28 16:54:35 operator.go:121: [info] [sql-operator] set a new operator - uuid: c959f2fb-f1c2-40c7-a1fa-e73cd51736dd, - pattern: ~(?i)ALTER\s+TABLE\s+`shard_db`.`shard_table`\s+DROP\s+COLUMN\s+`c2`, - op: REPLACE, args: ALTER TABLE `shard_db`.`shard_table` DROP INDEX idx_c2; ALTER TABLE `shard_db`.`shard_table` DROP COLUMN `c2` - on replication unit - ``` - - ```bash - 2018/12/28 16:54:35 operator.go:158: [info] [sql-operator] - sql-pattern ~(?i)ALTER\s+TABLE\s+`shard_db`.`shard_table`\s+DROP\s+COLUMN\s+`c2` matched SQL - USE `shard_db`; ALTER TABLE `shard_db`.`shard_table` DROP COLUMN `c2`;, - applying operator uuid: c959f2fb-f1c2-40c7-a1fa-e73cd51736dd, - pattern: ~(?i)ALTER\s+TABLE\s+`shard_db`.`shard_table`\s+DROP\s+COLUMN\s+`c2`, - op: REPLACE, args: ALTER TABLE `shard_db`.`shard_table` DROP INDEX idx_c2; ALTER TABLE `shard_db`.`shard_table` DROP COLUMN `c2` - ``` - - In addition, you can view the following log in the **DM-master** node: - - ```bash - 2018/12/28 16:54:35 operator.go:122: [info] [sql-operator] get an operator - uuid: eba35acd-6c5e-4bc3-b0b0-ae8bd1232351, request: name:"test" op:REPLACE - args:"ALTER TABLE `shard_db`.`shard_table` DROP INDEX idx_c2;" - args:"ALTER TABLE `shard_db`.`shard_table` DROP COLUMN `c2`" - sqlPattern:"~(?i)ALTER\\s+TABLE\\s+`shard_db`.`shard_table`\\s+DROP\\s+COLUMN\\s+`c2`" - sharding:true - with key ~(?i)ALTER\s+TABLE\s+`shard_db`.`shard_table`\s+DROP\s+COLUMN\s+`c2` matched SQL - USE `shard_db`; ALTER TABLE `shard_db`.`shard_table` DROP COLUMN `c2`; - ``` - - ```bash - 2018/12/28 16:54:36 operator.go:145: [info] [sql-operator] remove an operator - uuid: eba35acd-6c5e-4bc3-b0b0-ae8bd1232351, request: name:"test" op:REPLACE - args:"ALTER TABLE `shard_db`.`shard_table` DROP INDEX idx_c2;" - args:"ALTER TABLE `shard_db`.`shard_table` DROP COLUMN `c2`" - sqlPattern:"~(?i)ALTER\\s+TABLE\\s+`shard_db`.`shard_table`\\s+DROP\\s+COLUMN\\s+`c2`" - sharding:true - ``` - -7. Use `query-status` to guarantee that the `stage` of the task has been sustained as `Running`, and there is no more DDL statement that is blocking the replication (`blockingDDLs`) and no more sharding group to be resolved (`unresolvedGroups`). - -8. Use `query-error` to guarantee that no DDL execution error exists. - -9. Use `show-ddl-locks` to guarantee that all DDL locks have been resolved. diff --git a/zh/handle-failed-sql-statements.md b/zh/handle-failed-sql-statements.md index 28dd3ac96..32fdd5d06 100644 --- a/zh/handle-failed-sql-statements.md +++ b/zh/handle-failed-sql-statements.md @@ -122,7 +122,7 @@ ERROR 8200 (HY000): Unsupported modify column: can't change decimal column preci {{< copyable "" >}} ```bash - » hanlde-error test skip + » handle-error test skip ``` ``` @@ -251,7 +251,7 @@ ALTER TABLE `shard_db_*`.`shard_table_*` CHARACTER SET LATIN1 COLLATE LATIN1_DAN {{< copyable "" >}} ```bash - » hanlde-error test skip + » handle-error test skip ``` ``` @@ -296,7 +296,7 @@ ALTER TABLE `shard_db_*`.`shard_table_*` CHARACTER SET LATIN1 COLLATE LATIN1_DAN {{< copyable "" >}} ```bash - » hanlde-error test skip + » handle-error test skip ``` ``` From 25a613e90b8f59971959c72aa741e6ed59c89c48 Mon Sep 17 00:00:00 2001 From: Keke Yi <40977455+yikeke@users.noreply.github.com> Date: Fri, 21 Aug 2020 15:06:55 +0800 Subject: [PATCH 2/6] Update en/handle-failed-sql-statements.md --- en/handle-failed-sql-statements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/en/handle-failed-sql-statements.md b/en/handle-failed-sql-statements.md index 2782697bf..8d6795bd3 100644 --- a/en/handle-failed-sql-statements.md +++ b/en/handle-failed-sql-statements.md @@ -35,7 +35,7 @@ The `query-status` command is used to query the current status of items such as The `handle-error` command is used to handle the failed SQL statements. -### 命令用法 +### Command usage ```bash » handle-error -h From 1e3b96c4b14c740b70515747209df4bf20d8b262 Mon Sep 17 00:00:00 2001 From: Keke Yi <40977455+yikeke@users.noreply.github.com> Date: Fri, 21 Aug 2020 15:27:19 +0800 Subject: [PATCH 3/6] Apply suggestions from code review Co-authored-by: GMHDBJD <35025882+GMHDBJD@users.noreply.github.com> --- en/handle-failed-sql-statements.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/en/handle-failed-sql-statements.md b/en/handle-failed-sql-statements.md index 8d6795bd3..227df7ebb 100644 --- a/en/handle-failed-sql-statements.md +++ b/en/handle-failed-sql-statements.md @@ -6,7 +6,7 @@ aliases: ['/docs/tidb-data-migration/dev/skip-or-replace-abnormal-sql-statements # Handle Failed SQL Statements -This document introduces how to handle failed SQL statements when you're using the TiDB Data Migration tool to migrate data.. +This document introduces how to handle failed SQL statements when you're using the TiDB Data Migration tool to migrate data. Currently, TiDB is not completely compatible with all MySQL syntax (see [the DDL statements supported by TiDB](https://pingcap.com/docs/dev/reference/mysql-compatibility/#ddl)). Therefore, when DM is replicating data from MySQL to TiDB and TiDB does not support the corresponding SQL statement, an error might occur and break the replication process. In this case, you can use the `handle-error` command of DM to resume the replication. @@ -275,7 +275,7 @@ Assume that it is acceptable in the actual production environment that this DDL } ``` -2. Execute the `query-status` command, and you can see the errors reported by the `shard_db_1`.`shard_table_1` table in MySQL instance 1 and the `shard_db_2`.`shard_table_1` table in MySQL instance 2: +2. Execute the `query-status` command, and you can see the errors reported by the `shard_db_1`.`shard_table_2` table in MySQL instance 1 and the `shard_db_2`.`shard_table_2` table in MySQL instance 2: ``` { From 801afe8628b0ce27b32f3e1592d6acb2c3ff7df8 Mon Sep 17 00:00:00 2001 From: Keke Yi <40977455+yikeke@users.noreply.github.com> Date: Fri, 21 Aug 2020 15:32:48 +0800 Subject: [PATCH 4/6] Update en/handle-failed-sql-statements.md --- en/handle-failed-sql-statements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/en/handle-failed-sql-statements.md b/en/handle-failed-sql-statements.md index 227df7ebb..6439a7266 100644 --- a/en/handle-failed-sql-statements.md +++ b/en/handle-failed-sql-statements.md @@ -629,7 +629,7 @@ You can replace this DDL statement with two equivalent DDL statements. The steps } ``` -2. Use `query-status ` to view the task status, and you can see the following errors reported by the `shard_db_1`.`shard_table_1` table in MySQL instance 1 and the `shard_db_2`.`shard_table_1` table in MySQL instance 2: +2. Use `query-status ` to view the task status, and you can see the following errors reported by the `shard_db_1`.`shard_table_2` table in MySQL instance 1 and the `shard_db_2`.`shard_table_2` table in MySQL instance 2: ``` { From f250e66a3231142d20f37a324ea4a5feff2edd3b Mon Sep 17 00:00:00 2001 From: Keke Yi <40977455+yikeke@users.noreply.github.com> Date: Fri, 21 Aug 2020 15:35:06 +0800 Subject: [PATCH 5/6] Apply suggestions from code review Co-authored-by: Lilian Lee --- en/handle-failed-sql-statements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/en/handle-failed-sql-statements.md b/en/handle-failed-sql-statements.md index 6439a7266..98a3799fd 100644 --- a/en/handle-failed-sql-statements.md +++ b/en/handle-failed-sql-statements.md @@ -71,7 +71,7 @@ Global Flags: + `binlog-pos`: - Flag parameter, string, `--binlog-pos` - - If it is not specified, DM automatically handles the currently failed SQL statement + - If it is not specified, DM automatically handles the currently failed SQL statement. - If it is specified, the skip operation is executed when `binlog-pos` matches with the position of the binlog event. The format is `binlog-filename:binlog-pos`, for example, `mysql-bin|000001.000003:3270`. - When the replication returns an error, the binlog position can be obtained from `position` in `startLocation` returned by `query-status`. From 2a10e195aacf3f9a446a71374e06eac14aec1361 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Fri, 21 Aug 2020 15:45:33 +0800 Subject: [PATCH 6/6] Update en/handle-failed-sql-statements.md --- en/handle-failed-sql-statements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/en/handle-failed-sql-statements.md b/en/handle-failed-sql-statements.md index 98a3799fd..9f152eee6 100644 --- a/en/handle-failed-sql-statements.md +++ b/en/handle-failed-sql-statements.md @@ -6,7 +6,7 @@ aliases: ['/docs/tidb-data-migration/dev/skip-or-replace-abnormal-sql-statements # Handle Failed SQL Statements -This document introduces how to handle failed SQL statements when you're using the TiDB Data Migration tool to migrate data. +This document introduces how to handle failed SQL statements when you're using the TiDB Data Migration (DM) tool to migrate data. Currently, TiDB is not completely compatible with all MySQL syntax (see [the DDL statements supported by TiDB](https://pingcap.com/docs/dev/reference/mysql-compatibility/#ddl)). Therefore, when DM is replicating data from MySQL to TiDB and TiDB does not support the corresponding SQL statement, an error might occur and break the replication process. In this case, you can use the `handle-error` command of DM to resume the replication.