ddl rollingback is freezed

[chinesezhaoyun]

Bug Report

1. Minimal reproduce step (Required)

Cancel ddl state is freezed.

Asktug DDL rollingback 阻塞
Asktug 批量执行DDL处于done状态

Step as follow. But I can't reproduce the issue.

Enable variables
SET GLOBAL tidb_enable_metadata_lock = 'ON' ;
SET GLOBAL tidb_ddl_enable_fast_reorg = 'ON' ;

1. add index on big table
2. cancel add index
3. scale-out a tidb node
4. scale-in a tidb node
5. restart tidb node

+--------+--------------------------+-------------------------------------+---------------------------+-----------------------+-----------+----------+-----------+---------------------+---------------------+---------------------+-------------+
| JOB_ID | DB_NAME                  | TABLE_NAME                          | JOB_TYPE                  | SCHEMA_STATE          | SCHEMA_ID | TABLE_ID | ROW_COUNT | CREATE_TIME         | START_TIME          | END_TIME            | STATE       |
+--------+--------------------------+-------------------------------------+---------------------------+-----------------------+-----------+----------+-----------+---------------------+---------------------+---------------------+-------------+
|  23333 | spider_company_entity    | spider_enterprise_regno_sd_info     | add index /* txn-merge */ | delete reorganization |     16417 |    16627 | 142555210 | 2023-07-21 14:43:55 | 2023-07-21 14:43:55 | NULL                | rollingback |
|  23335 | spider_company_entity    | spider_enterprise_regno_sd_gt_info  | add index /* txn-merge */ | delete reorganization |     16417 |    16651 |  10023574 | 2023-07-21 14:59:12 | 2023-07-21 14:59:12 | NULL                | rollingback |
|  23357 | spider_company_entity    | spider_enterprise_regno_sd_info     | add index                 | none                  |     16417 |    16627 |         0 | 2023-07-21 16:26:56 | NULL                | NULL                | cancelling  |
|  24256 | spider_company_entity    | spider_enterprise_regno_sd_gt_info  | rename table              | none                  |     16417 |    16651 |         0 | 2023-08-01 18:14:06 | NULL                | NULL                | cancelling  |
|  24316 | spider_company_test      | t_proposed_notice_structural_middle | modify column             | public                |      7200 |    23199 |         0 | 2023-08-02 11:09:29 | 2023-08-02 11:09:29 | 2023-08-02 11:09:29 | synced      |
|  24315 | spider_company_base_info | t_proposed_notice_structural_middle | modify column             | public                |     13300 |    24191 |         0 | 2023-08-02 11:09:14 | 2023-08-02 11:09:14 | 2023-08-02 11:09:15 | synced      |
+--------+--------------------------+-------------------------------------+---------------------------+-----------------------+-----------+----------+-----------+---------------------+---------------------+---------------------+-------------+

tidb log

[2023/08/02 13:53:01.117 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id e8f6e8b5-c2fa-48c1-94e5-89094a9b7729"] ["ddl id"=23335] [ver=17987]
[2023/08/02 13:53:01.139 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id 82ad37d5-2f61-4616-af37-47cc32984a59"] ["ddl id"=23335] [ver=17987]
[2023/08/02 13:53:01.139 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id 82ad37d5-2f61-4616-af37-47cc32984a59"] ["ddl id"=23333] [ver=17988]
[2023/08/02 13:53:01.139 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id e8f6e8b5-c2fa-48c1-94e5-89094a9b7729"] ["ddl id"=23333] [ver=17988]
[2023/08/02 13:53:01.139 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id e8f6e8b5-c2fa-48c1-94e5-89094a9b7729"] ["ddl id"=23335] [ver=17987]
[2023/08/02 13:53:01.161 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id 82ad37d5-2f61-4616-af37-47cc32984a59"] ["ddl id"=23335] [ver=17987]
[2023/08/02 13:53:01.161 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id e8f6e8b5-c2fa-48c1-94e5-89094a9b7729"] ["ddl id"=23335] [ver=17987]
[2023/08/02 13:53:01.161 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id e8f6e8b5-c2fa-48c1-94e5-89094a9b7729"] ["ddl id"=23333] [ver=17988]
[2023/08/02 13:53:01.161 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id 82ad37d5-2f61-4616-af37-47cc32984a59"] ["ddl id"=23333] [ver=17988]
[2023/08/02 13:53:01.183 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id 82ad37d5-2f61-4616-af37-47cc32984a59"] ["ddl id"=23333] [ver=17988]
[2023/08/02 13:53:01.183 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id e8f6e8b5-c2fa-48c1-94e5-89094a9b7729"] ["ddl id"=23333] [ver=17988]
[2023/08/02 13:53:01.183 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id 82ad37d5-2f61-4616-af37-47cc32984a59"] ["ddl id"=23335] [ver=17987]
[2023/08/02 13:53:01.183 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id e8f6e8b5-c2fa-48c1-94e5-89094a9b7729"] ["ddl id"=23335] [ver=17987]
[2023/08/02 13:53:01.213 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id e8f6e8b5-c2fa-48c1-94e5-89094a9b7729"] ["ddl id"=23335] [ver=17987]
[2023/08/02 13:53:01.213 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id 82ad37d5-2f61-4616-af37-47cc32984a59"] ["ddl id"=23335] [ver=17987]
[2023/08/02 13:53:01.213 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id 82ad37d5-2f61-4616-af37-47cc32984a59"] ["ddl id"=23333] [ver=17988]
[2023/08/02 13:53:01.213 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id e8f6e8b5-c2fa-48c1-94e5-89094a9b7729"] ["ddl id"=23333] [ver=17988]

2. What did you expect to see? (Required)

ddl state is synced.

3. What did you see instead (Required)

Create new table and load data.

4. What is your TiDB version? (Required)

Release Version: v6.5.3
Edition: Community
Git Commit Hash: 71e6696
Git Branch: heads/refs/tags/v6.5.3
UTC Build Time: 2023-06-12 13:52:32
GoVersion: go1.19.9
Race Enabled: false
TiKV Min Version: 6.2.0-alpha
Check Table Before Drop: false
Store: tikv

[wjhuang2016]

@chinesezhaoyun Hi, Can it recover after some minutes?

[chinesezhaoyun]

@wjhuang2016 It still hasn't recovered. How can I fix it.

[wjhuang2016]

@wjhuang2016 It still hasn't recovered. How can I fix it.

@chinesezhaoyun
I replied to you in Asktug 批量执行DDL处于done状态, we can share the information there.

[D3Hunter]

see timeline in #57003, seems similar issue