server, core: support add learner node #896

Connor1996 · 2017-12-29T01:11:50Z

this pr does:

if enable raft learner, pd use AddLearnerPeer and PromoteLearnerPeer to replace AddNode
scheduler will not select region that has learners
to consider recovering from unfinished operator, pd check region that has learners but with no operator, and then pd will add PromoteLearnerPeer operator for it.

note that: peers = voters+learners

disksing · 2018-01-02T03:06:43Z

server/config.go

@@ -333,6 +333,8 @@ type ScheduleConfig struct {
 	ReplicaScheduleLimit uint64 `toml:"replica-schedule-limit,omitempty" json:"replica-schedule-limit"`
 	// TolerantSizeRatio is the ratio of buffer size for balance scheduler.
 	TolerantSizeRatio float64 `toml:"tolerant-size-ratio,omitempty" json:"tolerant-size-ratio"`
+	// EnableRaftLearner is the switch for using AddLearnerNode instead of AddNode


switch -> option.

disksing · 2018-01-02T03:11:12Z

server/coordinator.go

@@ -472,6 +472,32 @@ func (c *coordinator) sendScheduleCommand(region *core.RegionInfo, step schedule
 			},
 		}
 		c.hbStreams.sendMsg(region, cmd)
+	case schedule.AddLearnerPeer:


Can we remove a learner peer? I believe we need this feature in some cases.

I think we can reuse RemovePeer

disksing · 2018-01-02T03:14:53Z

server/core/region.go

+	DownPeers            []*pdpb.PeerStats
+	PendingPeers         []*metapb.Peer
+	PendingLearnerPeers  []*metapb.Peer
+	CompleteLearnerPeers []*metapb.Peer


Any document about how the CompleteLearner works? Specifically, when will a peer be added to the list and when will it be removed?

disksing · 2018-01-02T03:17:20Z

server/schedule/operator.go

+
+// Influence calculates the store difference that current step make
+func (plp PromoteLearnerPeer) Influence(opInfluence OpInfluence, region *core.RegionInfo) {
+	return


You can just leave the method body empty.

disksing · 2018-01-02T03:22:22Z

server/schedule/operator.go

+// IsFinish checks if current step is finished.
+func (alp AddLearnerPeer) IsFinish(region *core.RegionInfo) bool {
+	if p := region.GetStorePeer(alp.ToStore); p != nil {
+		return region.GetPendingLearnerPeer(p.GetId()) == nil && region.GetCompleteLearnerPeer(p.GetId()) != nil


AFAIU, a learner peer will be included in the peers list. This fact will affect some old utilities, such as RandomFollower. We should not select a learner as follower in some cases, for example we cannot transfer leader to a learner peer.

disksing · 2018-01-02T03:26:07Z

We should consider how to recover from unfinished operator. If pd's leader is changed, we need to remove a learner or promote it.

Connor1996 · 2018-02-08T08:01:37Z

PTAL @disksing @nolouch

siddontang · 2018-04-02T13:57:49Z

server/core/region.go

+}
+
+// ClassifyVoterAndLearner sorts out voter and learner from peers into different slice.
+func ClassifyVoterAndLearner(region *RegionInfo) {


do we need to export this function?

disksing · 2018-04-02T12:25:03Z

server/config.go

@@ -392,6 +395,7 @@ const (
 	defaultReplicaScheduleLimit = 32
 	defaultMergeScheduleLimit   = 20
 	defaultTolerantSizeRatio    = 2.5
+	defaultEnableRaftLearner    = false


remote this unused variable.

disksing · 2018-04-03T02:08:01Z

server/core/region.go

@@ -119,6 +157,26 @@ func (r *RegionInfo) GetDownPeer(peerID uint64) *metapb.Peer {
 	return nil
 }

+// GetDownVoter returns the down peer with specified peer id.


returns the down voter

disksing · 2018-04-03T02:28:09Z

https://github.com/pingcap/pd/blob/e221ffb59f7b3433ee2e0a617f616dc92b02d007/server/schedule/replica_checker.go#L188
We need to check if the region has any learners here. Consider maxReplica=1, a region has a offline leader and a learner, the leader will be removed.

disksing · 2018-04-03T03:02:46Z

https://github.com/pingcap/pd/blob/e221ffb59f7b3433ee2e0a617f616dc92b02d007/server/schedule/merge_checker.go#L40-L43
Need to make sure there is no learner peer.

disksing · 2018-04-04T06:46:42Z

LGTM.

nolouch · 2018-04-04T08:21:03Z

server/core/region.go

+	followers       map[uint64]*regionMap // storeID -> regionID -> regionInfo
+	learners        map[uint64]*regionMap // storeID -> regionID -> regionInfo
+	pendingPeers    map[uint64]*regionMap // storeID -> regionID -> regionInfo
+	pendingLearners map[uint64]*regionMap // storeID -> regionID -> regionInfo


remove pendingLearners

nolouch

LGTM

hicqu · 2018-04-11T12:13:14Z

server/schedule/operator.go

@@ -79,7 +79,7 @@ func (ap AddPeer) IsFinish(region *core.RegionInfo) bool {
 			log.Warnf("expect %v, but obtain voter %v", ap.String(), p.GetId())
 			return false
 		}
-		return region.GetPendingVoter(p.GetId()) == nil && p.GetId() != ap.PeerID
+		return region.GetPendingVoter(p.GetId()) == nil


Because AddNode won't have any subsequence operators?

I think above if block already checked if their IDs are matched, so we don't need it here.

Connor1996 · 2018-04-13T07:34:05Z

/run-all-tests

disksing · 2018-04-13T08:23:16Z

LGTM.

support add learner node

3e8ce83

Connor1996 changed the title ~~server, core: support add learner node~~ [WIP]server, core: support add learner node Dec 29, 2017

disksing reviewed Jan 2, 2018

View reviewed changes

fix typo

1f8a287

Connor1996 added 12 commits January 17, 2018 14:37

take learner peers into consideration when schedule

d1cc8e0

Merge branch 'master' into learner-node

35f9592

fix format

498cd22

address comment

eefe137

fix

e8dcf37

update kvproto

11439e0

fix format

dd6fd83

fix test

8b6aafa

Merge branch 'master' into learner-node

22ac86c

Merge branch 'master' into learner-node

93c3029

adjust

250a703

check learners

d3749ec

Connor1996 added 2 commits February 8, 2018 16:29

Merge branch 'master' into learner-node

2511bff

tiny change

48c5e88

Connor1996 force-pushed the learner-node branch from 998e14b to 48c5e88 Compare February 9, 2018 03:21

Connor1996 changed the title ~~[WIP]server, core: support add learner node~~ [DNM]server, core: support add learner node Feb 9, 2018

Connor1996 added 8 commits February 9, 2018 15:42

tiny change

9cef7a3

fix json unmarshal error

d2657a4

use AddNode instead of PromoteLearnerNode

1e97221

Merge branch 'master' into learner-node

bdefe62

Merge branch 'master' into learner-node

0daba6e

fix clone

ebdd942

Merge branch 'master' into learner-node

85fb534

update kvproto

e6213ad

Connor1996 added 3 commits April 2, 2018 17:44

Merge branch 'master' into learner-node

9ae33e5

remove pending learner count

90d232e

remove learner statistics

85965a9

siddontang reviewed Apr 2, 2018

View reviewed changes

disksing reviewed Apr 3, 2018

View reviewed changes

address comment

a3dc83c

Connor1996 force-pushed the learner-node branch from 1239f49 to a3dc83c Compare April 4, 2018 03:33

fix admin merge command

654fa8e

nolouch reviewed Apr 4, 2018

View reviewed changes

tiny clean up

772d4d4

nolouch approved these changes Apr 4, 2018

View reviewed changes

Connor1996 added 2 commits April 11, 2018 11:47

validity check

a728419

Merge branch 'master' into learner-node

6adc7e7

Connor1996 force-pushed the learner-node branch from 8173c88 to 6adc7e7 Compare April 11, 2018 03:48

fix

7553410

hicqu reviewed Apr 11, 2018

View reviewed changes

disksing added the priority/P1 The issue has P1 priority. label Apr 12, 2018

Connor1996 added 3 commits April 12, 2018 14:48

fix test

bb0fc2e

update kvproto

9b69953

Merge branch 'master' into learner-node

ef7ebe8

Connor1996 force-pushed the learner-node branch from 99737d8 to ef7ebe8 Compare April 13, 2018 07:31

disksing added the status/tests-passed The PR has passed all tests. label Apr 13, 2018

disksing merged commit 2c8e7d7 into tikv:master Apr 13, 2018

Connor1996 deleted the learner-node branch July 19, 2018 11:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

server, core: support add learner node #896

server, core: support add learner node #896

Connor1996 commented Dec 29, 2017 •

edited

Loading

disksing Jan 2, 2018

disksing Jan 2, 2018

Connor1996 Jan 17, 2018

disksing Jan 2, 2018

disksing Jan 2, 2018

disksing Jan 2, 2018

disksing commented Jan 2, 2018

Connor1996 commented Feb 8, 2018

siddontang Apr 2, 2018

disksing Apr 2, 2018

disksing Apr 3, 2018

disksing commented Apr 3, 2018 •

edited

Loading

disksing commented Apr 3, 2018

disksing commented Apr 4, 2018

nolouch Apr 4, 2018

nolouch left a comment

hicqu Apr 11, 2018

disksing Apr 11, 2018

Connor1996 commented Apr 13, 2018

disksing commented Apr 13, 2018

server, core: support add learner node #896

server, core: support add learner node #896

Conversation

Connor1996 commented Dec 29, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

disksing commented Jan 2, 2018

Connor1996 commented Feb 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

disksing commented Apr 3, 2018 • edited Loading

disksing commented Apr 3, 2018

disksing commented Apr 4, 2018

Choose a reason for hiding this comment

nolouch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Connor1996 commented Apr 13, 2018

disksing commented Apr 13, 2018

Connor1996 commented Dec 29, 2017 •

edited

Loading

disksing commented Apr 3, 2018 •

edited

Loading