Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make balance works #881

Merged
merged 14 commits into from
Sep 27, 2019
Merged

Make balance works #881

merged 14 commits into from
Sep 27, 2019

Conversation

dangleptr
Copy link
Contributor

@dangleptr dangleptr commented Sep 2, 2019 •
edited
Loading

subtask of #182
Main changes:

  1. Support raft member change. To be simple, we just add/remove one peer each time.
  2. Make balance works.
  3. Show specified balance plan in detail.
  4. Fix some problems when recovery.

Show the usage with an example:
Originally, the cluster have 7 hosts with 3 parts, 3 replica.
Let's scale the cluster to 9 hosts (one part one host).

  1. Start two new hosts directly. (192.168.8.210:44920, 192.168.8.210:44930)

  2. We could show our parts allocation with "show hosts" (3 parts, 3 replica, 7 hosts, 2 new hosts)

(user@127.0.0.1) [(none)]> show hosts
=================================================================================================
| Ip            | Port  | Status  | Leader count | Leader distribution | Partition distribution |
=================================================================================================
| 192.168.8.210 | 34600 | online  | 0            |                     | space 1: 1             |
-------------------------------------------------------------------------------------------------
| 192.168.8.210 | 34900 | online  | 1            | space 1: 1          | space 1: 1             |
-------------------------------------------------------------------------------------------------
| 192.168.8.210 | 35940 | online  | 0            |                     | space 1: 1             |
-------------------------------------------------------------------------------------------------
| 192.168.8.210 | 34920 | online  | 0            |                     | space 1: 1             |
-------------------------------------------------------------------------------------------------
| 192.168.8.210 | 44920 | online  | 0            |                     |                        |
-------------------------------------------------------------------------------------------------
| 192.168.8.210 | 44930 | online  | 0            |                     |                        |
-------------------------------------------------------------------------------------------------
| 192.168.8.210 | 34700 | online  | 1            | space 1: 1          | space 1: 1             |
-------------------------------------------------------------------------------------------------
| 192.168.8.210 | 34500 | online  | 0            |                     | space 1: 2             |
-------------------------------------------------------------------------------------------------
| 192.168.8.210 | 34800 | online  | 1            | space 1: 1          | space 1: 2             |
-------------------------------------------------------------------------------------------------
  1. Now, we could use balance as follows, balance id will be showed in table.
(user@127.0.0.1) [(none)]> balance data
==============
| ID         |
==============
| 1568100011 |
--------------
Got 1 rows (Time spent: 3483/4713 us)
  1. The balance has been runned in the backend. Now, we could use "balance data $id" to show the detail of the plan.
(user@127.0.0.1) [(none)]> balance data 1568100011
==============================================================================
| balanceId, spaceId:partId, src->dst                          | status      |
==============================================================================
| [1568100011, 1:1, 192.168.8.210:34800->192.168.8.210:44930]  | in progress |
------------------------------------------------------------------------------
| [1568100011, 1:2, 192.168.8.210:34500->192.168.8.210:44920]  | in progress |
------------------------------------------------------------------------------
Got 2 rows (Time spent: 1273/2097 us)
  1. After finishing the balance plan, we could check the status with "show hosts"
(user@127.0.0.1) [(none)]> show hosts
=================================================================================================
| Ip            | Port  | Status  | Leader count | Leader distribution | Partition distribution |
=================================================================================================
| 192.168.8.210 | 34600 | online  | 0            |                     | space 1: 1             |
-------------------------------------------------------------------------------------------------
| 192.168.8.210 | 34900 | online  | 1            | space 1: 1          | space 1: 1             |
-------------------------------------------------------------------------------------------------
| 192.168.8.210 | 35940 | online  | 0            |                     | space 1: 1             |
-------------------------------------------------------------------------------------------------
| 192.168.8.210 | 34920 | online  | 1            | space 1: 1          | space 1: 1             |
-------------------------------------------------------------------------------------------------
| 192.168.8.210 | 44920 | online  | 0            |                     | space 1: 1             |
-------------------------------------------------------------------------------------------------
| 192.168.8.210 | 44930 | online  | 0            |                     | space 1: 1             |
-------------------------------------------------------------------------------------------------
| 192.168.8.210 | 34700 | online  | 1            | space 1: 1          | space 1: 1             |
-------------------------------------------------------------------------------------------------
| 192.168.8.210 | 34500 | online  | 0            |                     | space 1: 1             |
-------------------------------------------------------------------------------------------------
| 192.168.8.210 | 34800 | online  | 0            |                     | space 1: 1             |
-------------------------------------------------------------------------------------------------

@dangleptr dangleptr changed the title Member change RAFT Member change Sep 2, 2019
@dangleptr dangleptr mentioned this pull request Sep 3, 2019
Merged
@dangleptr dangleptr added the ready-for-testing PR: ready for the CI test label Sep 3, 2019
@nebula-community-bot
Copy link
Member

Unit testing passed.

@nebula-community-bot
Copy link
Member

Unit testing failed.

@nebula-community-bot
Copy link
Member

Unit testing failed.

@nebula-community-bot
Copy link
Member

Unit testing passed.

@dangleptr dangleptr changed the title RAFT Member change Make balance works Sep 10, 2019
@nebula-community-bot
Copy link
Member

Unit testing failed.

@nebula-community-bot
Copy link
Member

Unit testing failed.

@nebula-community-bot
Copy link
Member

Unit testing passed.

wadeliuyi
wadeliuyi reviewed Sep 10, 2019
View reviewed changes
src/graph/BalanceExecutor.cpp
}
auto tasks = std::move(resp).value();
resp_ = std::make_unique<cpp2::ExecutionResponse>();
std::vector<std::string> header{"balanceId, spaceId:partId, src->dst", "status"};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can create header in client directly?

@dangleptr dangleptr removed the ready-for-testing PR: ready for the CI test label Sep 17, 2019
@dangleptr dangleptr added the ready-for-testing PR: ready for the CI test label Sep 19, 2019
@dangleptr
Copy link
Contributor Author

jenkins go

@nebula-community-bot
Copy link
Member

Unit testing failed.

1 similar comment
@nebula-community-bot
Copy link
Member

Unit testing failed.

@dangleptr dangleptr mentioned this pull request Sep 22, 2019
Closed
@nebula-community-bot
Copy link
Member

Unit testing passed.

@nebula-community-bot
Copy link
Member

Unit testing passed.

@nebula-community-bot
Copy link
Member

Unit testing passed.

@nebula-community-bot
Copy link
Member

Unit testing passed.

critical27
critical27 approved these changes Sep 26, 2019
View reviewed changes
Copy link
Contributor

@critical27 critical27 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

@dutor dutor merged commit 0ec5d7c into vesoft-inc:master Sep 27, 2019
@dangleptr dangleptr deleted the member-change branch September 28, 2019 04:30
@amber-moe
Copy link
Contributor

doc pr #1037

@amber-moe amber-moe mentioned this pull request Oct 10, 2019
Merged
yixinglu pushed a commit to yixinglu/nebula that referenced this pull request Feb 16, 2020
@dangleptr @dutor
Make balance works (vesoft-inc#881)
4224999 
* Implement member change

* Address wadeliuyi's comments

* Add Balance UT

* Fix failed Meta http Uts

* Add balance data and balance data id command in console

* Fix bugs

* Fix stale peers when add part

* Fix bugs

* Remove retry logic and fix stale log response

* Address cirtical27's comments

* Fix snapshot_test

* Fix bug in snapshot

* Remove check in commitTransferLeader
tong-hao pushed a commit to tong-hao/nebula that referenced this pull request Jun 1, 2021
@dangleptr @dutor
Make balance works (vesoft-inc#881)
fd76aef 
* Implement member change

* Address wadeliuyi's comments

* Add Balance UT

* Fix failed Meta http Uts

* Add balance data and balance data id command in console

* Fix bugs

* Fix stale peers when add part

* Fix bugs

* Remove retry logic and fix stale log response

* Address cirtical27's comments

* Fix snapshot_test

* Fix bug in snapshot

* Remove check in commitTransferLeader
yixinglu pushed a commit to yixinglu/nebula that referenced this pull request Jan 31, 2023
@nebula-bots @Shylock-Hg @Sophie-Xie
Move input rows of Traverse and AppendVertices. (vesoft-inc#881)
ba40fe5 
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

Co-authored-by: shylock <33566796+Shylock-Hg@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wadeliuyi wadeliuyi wadeliuyi left review comments

@darionyaphet darionyaphet darionyaphet approved these changes

@critical27 critical27 critical27 approved these changes

@sherman-the-tank sherman-the-tank Awaiting requested review from sherman-the-tank

Assignees
No one assigned
Labels
ready-for-testing PR: ready for the CI test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@dangleptr @nebula-community-bot @amber-moe @darionyaphet @critical27 @wadeliuyi @dutor