TiKV stores contains more than 1000 empty regions respectively after one importing, which blocks the next import

[Yui-Song]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Lauch a T3 TiDB Cloud cluster
2. Import TPCC 5000 warehouse data from s3://dbaas-benchmark-2/tpcc5k-parquet/
3. import sysbench data from s3://dbaas-benchmark-1/sysbench-table16-10m-parquet/

2. What did you expect to see? (Required)

All the dataset was imported successfully.

3. What did you see instead (Required)

The error message was too long that the most important part wasn't show on the console. The user just didn't know what's going wrong and got confused.

Log:

[2021/11/22 11:51:06.094 +00:00] [ERROR] [lightning.go:219] ["tidb lightning encountered error"] [error="tidb-lightning check failed. Please fix the failed check(s):\n Source csv files size is proper;\ncheckpoints are valid;\ntable schemas are valid;\nCluster is available;\nLightning has the correct storage permission;\nlocal disk resources are rich, estimate sorted data size 18.45GiB, local available is 2.953TiB;\nCluster available is rich, available is 2.153TiB, we need 55.34GiB;\nTiKV stores (5, 1, 4) contains more than 1000 empty regions respectively, which will greatly affect the import speed and success rate;\nCluster region distribution is balanced"] [errorVerbose="tidb-lightning check failed. Please fix the failed check(s):\n Source csv files size is proper;\ncheckpoints are valid;\ntable schemas are valid;\nCluster is available;\nLightning has the correct storage permission;\nlocal disk resources are rich, estimate sorted data size 18.45GiB, local available is 2.953TiB;\nCluster available is rich, available is 2.153TiB, we need 55.34GiB;\nTiKV stores (5, 1, 4) contains more than 1000 empty regions respectively, which will greatly affect the import speed and success rate;\nCluster region distribution is balanced\ngithub.com/pingcap/tidb/br/pkg/lightning/restore.(*Controller).preCheckRequirements\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/restore/restore.go:1929\ngithub.com/pingcap/tidb/br/pkg/lightning/restore.(*Controller).Run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/restore/restore.go:446\ngithub.com/pingcap/tidb/br/pkg/lightning.(*Lightning).run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/lightning.go:331\ngithub.com/pingcap/tidb/br/pkg/lightning.(*Lightning).RunServer\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/lightning.go:216\nmain.main.func2\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/tidb-lightning/main.go:80\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/tidb-lightning/main.go:87\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"]

4. What is your TiDB version? (Required)

Lightning version 5.3.0

| Release Version: v5.2.1
Edition: Community
Git Commit Hash: cd8fb24c5f7ebd9d479ed228bb41848bd5e97445
Git Branch: heads/refs/tags/v5.2.1
UTC Build Time: 2021-09-08 02:32:56
GoVersion: go1.16.4
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false |

[fubinzh]

TiKV side issue opened: tikv/tikv#11470

[glorv]

The max region keys in lightning is 1440000, but this will trigger tikv auto-split. Lightning should decrease this threshold to avoid region split after import.

[github-actions]

Please check whether the issue should be labeled with 'affects-x.y' or 'fixes-x.y.z', and then remove 'needs-more-info' label.