persist learner info

[pengweisong]

What type of PR is this?

• bug
• feature
• enhancement

What problem(s) does this PR solve?

Issue(s) number:
#3689

Description:
Balance is a long process. If the storaged restart when the cluster is doing balancing, we will lose some info now.
That is, all the learner info will be lost, because they will neither be persisted in storaged nor in metad.

How do you solve it?

Now the partition balance process includes:

1. ADD_PART_ON_DST: create a new part in target storaged.
2. ADD_LEARNER: add the new part as a raft learner in all partition's peers.
3. CATCH_UP_DATA: wait the new part catching data.
4. MEMBER_CHANGE_ADD: promote the new part from learner as follower.
5. MEMBER_CHANGE_REMOVE: remove the old part from all peers.
6. UPDATE_PART_META: update part hosts info in the metad.
7. REMOVE_PART_ON_SRC: remove unused data in the source storaged.
8. CHECK: check the part is good.

We will persist all the partition peers info in the storage local, including the status in balancing.
When the storaged restart, we will join the info in storage local with the meta to decide if the part should be kept or removed, started as learner or normal peer, started with which peers.

Special notes for your reviewer, ex. impact of this fix, design document, etc:

Checklist:

Tests:

• Unit test(positive and negative cases)
• Function test
• Performance test
• N/A

Affects:

• Documentation affected (Please add the label if documentation needs to be modified.)
• Incompatibility (If it breaks the compatibility, please describe it and add the label.）
• If it's needed to cherry-pick (If cherry-pick to some branches is required, please label the destination version(s).)
• Performance impacted: Consumes more CPU/Memory

Release notes:

Please confirm whether to be reflected in release notes and how to describe:

ex. Fixed the bug .....

[wenhaocs]

Awesome job!

[critical27]

Good job~ You'd better write a simple test case: write some learner info, restart the NebulaStore, check if the state is correct. I concern about the convert between storage address and raft address.

[panda-sheep]

Good job！

[liwenhui-soul]

why don't you merge them into one commit, it's hard to read

[liwenhui-soul]

did you consider about removing parts in metaclient?

[pengweisong]

@liwenhui-soul Metaclient only calculate diff in two version of local cache, but when add peer, we will not update local cache.

[critical27]

Generally LGTM~

[critical27]

Well done... This bug fix is way more complicated than I expected.