Lightning retry too much when tikv storage is full

[fubinzh]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Use lightning to import 60K tables to tidb cluster, TiKV space was full during import process

2. What did you expect to see? (Required)

1. Lightning should fail quickly when tikv doesn't have enough space.

3. What did you see instead (Required)

1. Lightning keep retrying for different tables for over 16 hours, before I have to manually stop it.

[root@zhengrong-24 lightning]# grep "The available disk of TiKV" tidb-lightning.log |grep 105.44 |head
[2022/03/23 23:01:27.971 +08:00] [WARN] [local.go:1153] ["write to tikv failed"] [error="The available disk of TiKV (172.16.105.44:50160) only left 392774759219, and capacity is 3937351139065"] [retry=0]
[2022/03/23 23:01:27.971 +08:00] [WARN] [local.go:1153] ["write to tikv failed"] [error="The available disk of TiKV (172.16.105.44:50160) only left 392774759219, and capacity is 3937351139065"] [retry=0]
[2022/03/23 23:01:27.971 +08:00] [WARN] [local.go:1153] ["write to tikv failed"] [error="The available disk of TiKV (172.16.105.44:50160) only left 392774759219, and capacity is 3937351139065"] [retry=0]
[2022/03/23 23:01:27.971 +08:00] [WARN] [local.go:1153] ["write to tikv failed"] [error="The available disk of TiKV (172.16.105.44:50160) only left 392774759219, and capacity is 3937351139065"] [retry=0]
[2022/03/23 23:01:27.971 +08:00] [WARN] [local.go:1153] ["write to tikv failed"] [error="The available disk of TiKV (172.16.105.44:50160) only left 392774759219, and capacity is 3937351139065"] [retry=0]
[2022/03/23 23:01:27.971 +08:00] [WARN] [local.go:1153] ["write to tikv failed"] [error="The available disk of TiKV (172.16.105.44:50160) only left 392774759219, and capacity is 3937351139065"] [retry=0]
[2022/03/23 23:01:27.971 +08:00] [WARN] [local.go:1153] ["write to tikv failed"] [error="The available disk of TiKV (172.16.105.44:50160) only left 392774759219, and capacity is 3937351139065"] [retry=0]
[2022/03/23 23:01:27.971 +08:00] [WARN] [local.go:1153] ["write to tikv failed"] [error="The available disk of TiKV (172.16.105.44:50160) only left 392774759219, and capacity is 3937351139065"] [retry=0]
[2022/03/23 23:01:27.972 +08:00] [WARN] [local.go:1153] ["write to tikv failed"] [error="The available disk of TiKV (172.16.105.44:50160) only left 392774759219, and capacity is 3937351139065"] [retry=1]
[2022/03/23 23:01:27.972 +08:00] [WARN] [local.go:1153] ["write to tikv failed"] [error="The available disk of TiKV (172.16.105.44:50160) only left 392774759219, and capacity is 3937351139065"] [retry=1]
[root@zhengrong-24 lightning]#
[root@zhengrong-24 lightning]#
[root@zhengrong-24 lightning]# grep "The available disk of TiKV" tidb-lightning.log |grep 105.44 |tail
[2022/03/24 16:28:05.357 +08:00] [ERROR] [restore.go:168] [-] [table=`Abetting_0`.`15unit_8_game_bets_all_analysis`] [status=imported] [error="[277a6aea-8948-5a38-ae13-b38e88c9a1ca] import reach max retry 3 and still failed: The available disk of TiKV (172.16.105.44:50160) only left 392452636672, and capacity is 3937351139065; The available disk of TiKV (172.16.105.44:50160) only left 392452636672, and capacity is 3937351139065"]
[2022/03/24 16:28:05.357 +08:00] [ERROR] [restore.go:168] [-] [table=`Abetting_0`.`14unit_0_game_bets_game_tag_analysis_1`] [status=imported] [error="[19832b46-1e76-5c2a-bfd9-1151f2234e9b] import reach max retry 3 and still failed: The available disk of TiKV (172.16.105.44:50160) only left 392452636672, and capacity is 3937351139065; The available disk of TiKV (172.16.105.44:50160) only left 392452636672, and capacity is 3937351139065"]
[2022/03/24 16:28:05.357 +08:00] [ERROR] [restore.go:168] [-] [table=`Abetting_0`.`12unit_2_game_bets_game_tag_analysis_2`] [status=imported] [error="[b414cffd-bcea-5e8b-a60f-fe0ed241c148] import reach max retry 3 and still failed: The available disk of TiKV (172.16.105.44:50160) only left 392452636672, and capacity is 3937351139065; The available disk of TiKV (172.16.105.44:50160) only left 392452636672, and capacity is 3937351139065"]

4. What is your TiDB version? (Required)

[root@zhengrong-24 lightning]# ./tidb-lightning -V
Release Version: v6.0.0
Git Commit Hash: e130e52
Git Branch: heads/refs/tags/v6.0.0
Go Version: go1.18
UTC Build Time: 2022-03-18 07:34:51
Race Enabled: false

[niubell]

/unassign gozssky

[niubell]

/assign D3Hunter