PD processes heartbeat slowly after scaling within a large cluster.

[nolouch]

Bug Report

PD processes heartbeat slowly after scaling within a large cluster.

After scaling, 7.1.x will estimate the progress of scale, the estimate will hold the lock too long.

pd/server/cluster/cluster.go

Lines 1943 to 1951 in 3f1a688

	for _, s := range stores {
	if s.IsRemoving() || s.IsRemoved() {
	continue
	}
	if placement.MatchLabelConstraints(s, rule.LabelConstraints) {
	matchStores = append(matchStores, s)
	}
	}
	regionSize := c.core.GetRegionSizeByRange(startKey, endKey) * int64(rule.Count)

Here GetRegionSizeByRange will iterate over many regions.

stack info in 7.1.1:

14708261976110 151990 @ 0xf0ada9 0x171c036 0x20d30b1 0x20d2bd6 0x20d171c 0x20c72d4 0xefb901
#	0xf0ada8	sync.(*RWMutex).RUnlock+0x28							/usr/local/go/src/sync/rwmutex.go:119
#	0x171c035	github.com/tikv/pd/pkg/core.(*RegionsInfo).GetRegionSizeByRange+0x175		/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/core/region.go:1538
#	0x20d30b0	github.com/tikv/pd/server/cluster.(*RaftCluster).calculateRange+0x2b0		/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/cluster/cluster.go:1851
#	0x20d2bd5	github.com/tikv/pd/server/cluster.(*RaftCluster).getThreshold+0x3f5		/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/cluster/cluster.go:1825
#	0x20d171b	github.com/tikv/pd/server/cluster.(*RaftCluster).checkStores+0x41b		/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/cluster/cluster.go:1752
#	0x20c72d3	github.com/tikv/pd/server/cluster.(*RaftCluster).runNodeStateCheckJob+0xd3	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/cluster/cluster.go:539

[rleungx]

It's a known problem, especially when the rule count is small.

[CabinfeverB]

It's a known problem, especially when the rule count is small.

when the rule count is small means the key range of a rule is too large?

[rleungx]

It's a known problem, especially when the rule count is small.

when the rule count is small means the key range of a rule is too large?

Yes

[seiya-annie]

/report customer