Sometimes failed to create scatter operator after splitting

[YuJuncen]

Bug Report

cc pingcap/tidb#31034

What did you do?

Execute ScatterRegions over some fresh regions created by BatchSplitRegion.

What did you expect to see?

The regions are scattered and balanced.

What did you see instead?

Some of the operators failed to be created because of unhealthy. And the final region isn't balanced.

What version of PD are you using (pd-server -V)?

/var/lib/pd/log # /pd-server -V
Release Version: v5.4.0-nightly
Edition: Community
Git Commit Hash: ae23d409c528a836ae6d98cd174689c38ad19f2d
Git Branch: heads/refs/tags/v5.4.0-nightly
UTC Build Time:  2021-12-30 12:11:23

Note

(The details of metric and cluster info TBD)

[YuJuncen]

What happened:

1. When we running restore, we found that the region is unbalanced.

2. Then, we have checked the PD dashboard, and found that there are many failure of scatter:

3. According to the DEBUG log, we found the reason of failed to create scatter operator is mainly unhealthy. But the cluster looks fine. (Those empty regions are freshly split and would be filled with data soon)

Note that this metric may not accurate enough.

4. We also find during scattering, there are many add-rule-peer operator created.

I'm wondering: What is the reason of that failure? How can I get why those add-rule-peer operator created, are they relative to the failure?

[nolouch]

I'm wondering: What is the reason of that failure? How can I get why those add-rule-peer operator created, are they relative to the failure?

Good question, I think we need do some works to improve the diagnosis. like #4418

[rleungx]

The problem of add-rule-peer is tracked by #4565 and it has been fixed. The reason why the operator fails is that the region has pending peers. We can do a check before starting the scatter region. See pingcap/tidb#31691.

Here is the test result of the above PR: