[Zone] Maintenance reminder for downsizing: Downsizing operation may fail, and manual handling of the "balance data" job task is required.

[jinyingsunny]

Please check the FAQ documentation before raising an issue
当part比较多 ，比如100，那么执行缩容的时候，可能因为任务失败而缩容中断，这时，需要人为处理失败的任务，比如recover job xxx；balance常见报错如下：

I20230918 02:26:21.756175   255 BalanceTask.cpp:81] 10, 5:48,nebulazone-storaged-15.nebulazone-storaged-headless.nebula.svc.cluster.local:9779->nebulazone-storaged-6.nebulazone-storaged-headless.nebula.svc.cluster.local:9779 Transfer leader failed, status Transfer leader failed: E_RETRY_EXHAUSTED

另外，因为缩容需要逐个space做balance data in zone remove，如果space比较多 ，出现上面👆🏻失败的概率会更高，用户需要找到存在balance data失败的space做recover处理。

可以通过kubectl -n <namespace> describe nc <cluster_name> 查看状态

(root@nebula) [王者荣耀]> show jobs
+--------+----------------+------------+----------------------------+----------------------------+
| Job Id | Command        | Status     | Start Time                 | Stop Time                  |
+--------+----------------+------------+----------------------------+----------------------------+
| 10     | "DATA_BALANCE" | "FAILED"   | 2023-09-18T02:25:51.000000 | 2023-09-18T02:26:37.000000 |
| 6      | "STATS"        | "FINISHED" | 2023-09-18T02:14:30.000000 | 2023-09-18T02:14:30.000000 |
+--------+----------------+------------+----------------------------+----------------------------+
Got 2 rows (time spent 7.862ms/8.791903ms)

Mon, 18 Sep 2023 02:29:52 UTC

(root@nebula) [王者荣耀]> desc space  王者荣耀
+----+------------+------------------+----------------+---------+------------+--------------------+-----------------------------------------------------+---------+
| ID | Name       | Partition Number | Replica Factor | Charset | Collate    | Vid Type           | Zones                                               | Comment |
+----+------------+------------------+----------------+---------+------------+--------------------+-----------------------------------------------------+---------+
| 5  | "王者荣耀" | 100              | 3              | "utf8"  | "utf8_bin" | "FIXED_STRING(32)" | ["us-central1-a", "us-central1-b", "us-central1-c"] |         |
+----+------------+------------------+----------------+---------+------------+--------------------+-----------------------------------------------------+---------+
Got 1 rows (time spent 6.739ms/13.160116ms)

Mon, 18 Sep 2023 02:43:43 UTC

Your Environments (required)

operator 镜像：1.6

[jinyingsunny]

mark as known issue