br restore database slow and lots of "scatter region failed, retring" in log when stores less then replica

[ChenPeng2013]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1 tidb, 1 pd, 1 tikv

use test;
// create many list partition tables;
BACKUP DATABASE test to 'local:///tmp/lp_backup/';
drop database test;
create database test;
use test;
RESTORE DATABASE * FROM 'local:///tmp/lp_backup/';

2. What did you expect to see? (Required)

restore success

3. What did you see instead (Required)

mysql> show processlist;
+------+------+----------------------+------+---------+-------+------------+---------------------------------------------------+
| Id   | User | Host                 | db   | Command | Time  | State      | Info                                              |
+------+------+----------------------+------+---------+-------+------------+---------------------------------------------------+
|   49 | root | 192.168.228.89:59988 | test | Query   | 12070 | autocommit | RESTORE DATABASE * FROM 'local:///tmp/lp_backup/' |
|   57 | root | 192.168.228.89:54784 | NULL | Query   |     0 | autocommit | show processlist                                  |
+------+------+----------------------+------+---------+-------+------------+---------------------------------------------------+
2 rows in set (0.00 sec)

4. What is your TiDB version? (Required)

Release Version: v5.2.0
Edition: Community
Git Commit Hash: d4d667fba5e82f108c62cd690fcf73c685d7612f
Git Branch: heads/refs/tags/v5.2.0
UTC Build Time: 2021-08-18 12:35:12
GoVersion: go1.16.4
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false

[YuJuncen]

Because of pingcap/br#1402, when a region hasn't been full replicated, BR would retry scatter until it was fully replicated.

However, unfortunately, when there aren't enough stores, this would always fail, and backoff time would become unreasonable. We can skip scatter in this condition to solve this.

[YuJuncen]

BTW, you can set max-replicas to 1 as a workaround.

pd-ctl config set max-replicas 1

[ti-srebot]

Please edit this comment or add a new comment to complete the following information

Not a bug

1. Remove the 'type/bug' label
2. Add notes to indicate why it is not a bug

Duplicate bug

1. Add the 'type/duplicate' label
2. Add the link to the original bug

Bug

Note: Make Sure that 'component', and 'severity' labels are added
Example for how to fill out the template: #20100

1. Root Cause Analysis (RCA) (optional)

2. Symptom (optional)

3. All Trigger Conditions (optional)

4. Workaround (optional)

5. Affected versions

6. Fixed versions