Upgrade TiDB without manually Canceling DDL

[zimulala]

Feature Request

In the documentation, we said do not use DDL statements when upgrading:

In addition, during the cluster upgrade, DO NOT execute any DDL statement. Otherwise, the issue of undefined behavior might occur

Details:
https://docs.pingcap.com/tidb/dev/upgrade-tidb-using-tiup#:~:text=DO%20NOT%20upgrade,behavior%20might%20occur.
We'd better add a feature to ensure this, not just documentation.

Development Plan

• Pause user DDL when upgrading TiDB design #43060
• Abstract out a SyncerWatch in syncer.go #43092
• Add global state syncer for handling data to etcd. #43112
• Add the functions of set/get the owner value #43352
• Using pause and resume DDL jobs: Add backend task pause|resume feature. #40041
• Update bootstrap and support pause user DDL when upgrading TiDB #43665
• Add mock upgrading code for testing #43418

User Doc

• *: add smooth upgrade TiDB introduction doc docs-cn#14013
• *: add a limit for smooth upgrade docs-cn#14052

TODO List

• Update hasSysDB using check ID instead of name or get all related DBs #43818
• Unify logs when upgrading #44017

Feature-related Issues

• DDL job not advancing after TiDB upgrade #43850
• Upgrade failure Re-upgrade may cause paused system DDL job can't be executed or dirty data of DDL job #44158
• slowly pause user ddl will let tidb upgrade failed #44159
• Before the upgrade, it has a paused DDL job, it's upgrade will fail #44225
• There are tasks to add indexes before and after the upgrade, and there is a problem with the order of processing on different tables #44304

Feature-unrelated Issues

• Kill TiDB can not support cancel DDL #43854

Future Works

1. Do not allow running DDL jobs to be processed on different versions of TiDB.
2. Consider not letting the new version of DDL jobs be processed on the old version of TiDB, and not letting the old version of DDL jobs be processed on the new version of TiDB.
3. Can not perform admin cancel DDL during the upgrade.