-
Notifications
You must be signed in to change notification settings - Fork 285
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(validator): elections powered by etcd leases (#31)
* feat(validator): elections powered by etcd leases Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com> * docs: etcd leases, state and deployment diagram Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com> * fix(quorum): docker-compose network name typo It was copy-pasted from the fabric docker-compose and so the network names were incorrectly referencing the fabric network, not the quorum. Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com> * refactor(validator): change getter to method The getter that was changed: isCurrentNodeLeader Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com> * refactor(validator): Object.assign -> line by line TODO(peter.somogyvari): Once leader election has enough test coverage, get rid of these property assignments and just use the this.leaderNodeInfo.networkInfo object directly everywhere. Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com> * fix(test/validator): isCurrentNodeLeader syntax Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com> * fix(examples,tools): use local npm package The examples were hardcoding GitHub branch references which is a moving target and makes it error prone in certain situations to have the intended code running for local tests/CI. Solved it with a pre-install npm script that runs npm pack in the root directory of the project creating an installable tarball of the latest source code that's checked out at the current moment. This strongly ties the examples of any given revision to the exact source code of BIF from that revision as well, guaranteeing that whoever/wherever does an npm install inside the examples will get the current source code of that revision packaged up as an npm dependency. that they are committed with Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com> * fix(example/quorum): container resource usage Reverting an attempted fix that was breaking the CI. The attempted fix was to introduce resource constraints on the quorum containers (CPU, RAM usage). This worked in the sense that on a smaller dev laptop launching the quorum network didn't cause OOM, but on the other hand containers got stuck in 'unhealthy' status on CI so overall the fix didn't pan out. Also: Performance optimization is added by reducing logging verbosity within the quorum containers and the geth syncmode is being changed from 'full' to 'fast' (the default) which does not verify each block one by one upon initialization. Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com> * fix(examples/corda): adds etcd to docker-compose This was forgotten from the original commit which only updated fabrid and quorum Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com> * docs(tools): add docs for create local npm packge Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
- Loading branch information
Showing
19 changed files
with
2,154 additions
and
475 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,3 @@ | ||
| node_modules | ||
| test | ||
| .tmp/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -32,3 +32,4 @@ typings/ | |
| examples/simple-asset-transfer/fabric/**/hfc-key-store/ | ||
|
|
||
| bin/ | ||
| .tmp/ | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| .github/ | ||
| .tmp/ | ||
| .vscode/ | ||
| bin/ | ||
| docs/ | ||
| examples/ | ||
| testes/ | ||
| tools/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| @startuml leader-election-etcd-leases-deployment-caption | ||
| !include <tupadr3/common> | ||
| !include <office/Servers/database_server> | ||
| !include <office/Servers/application_server> | ||
|
|
||
| center header | ||
| Hyperledger Blockchain Integration Framework | ||
|
|
||
| endheader | ||
|
|
||
| title | ||
| <u>Leader Election Deployment Diagram</u> | ||
|
|
||
| end title | ||
|
|
||
| center footer Hyperledger Blockchain Integration Framework, 2019 | ||
|
|
||
| frame BIF { | ||
|
|
||
| frame Etcd_Cluster as ec { | ||
| OFF_DATABASE_SERVER(etcd1,"Etcd 1") | ||
| OFF_DATABASE_SERVER(etcd2,"Etcd 2") | ||
| OFF_DATABASE_SERVER(etcdn,"Etcd N") | ||
| } | ||
|
|
||
| frame Validator_Cluster as vc { | ||
| OFF_APPLICATION_SERVER(bvn1,"Validator 1") | ||
| OFF_APPLICATION_SERVER(bvn2,"Validator 2") | ||
| OFF_APPLICATION_SERVER(bvnn,"Validator N") | ||
| } | ||
|
|
||
| } | ||
|
|
||
| bvn2 <~~> bvnn | ||
| bvn1 <~> bvn2 | ||
| bvn1 <~> bvnn | ||
|
|
||
| etcd1 <~~> etcd2 | ||
| etcd1 <~> etcdn | ||
| etcd2 <~> etcdn | ||
|
|
||
| bvn1 <=[#blue]=> ec | ||
| bvn2 <=[#blue]=> ec | ||
| bvnn <=[#blue]=> ec | ||
|
|
||
| @enduml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| # Leader Election via Etcd's Distributed Synchronization Primitives | ||
|
|
||
| ## Summary | ||
|
|
||
| Validators nodes self-elect a leader by leveraging the Etcd API's `watch` and `lease` features. BIF depends on Etcd API version 3, but other than that it's able to work with any Etcd cluster or single node configuration as long as connectivity is achievable. | ||
|
|
||
| At startup, each node attempts to make themselves leader, which will only succeed if there is no current leader. | ||
|
|
||
| If a validator node fails to become leader, it starts acting as a follower. | ||
| The identity of the leader node is stored under a specific key in Etcd on which the leader holds the lease. | ||
|
|
||
| Followers are watching the key storing the leaders identity and react to changes as soon as the change is detected, there is no polling that would waste network/compute resources. | ||
|
|
||
| Stability of the leader election is guaranteed by the Raft algorithm that Etcd uses under the hood. | ||
|
|
||
| ## Deployment Diagram | ||
|
|
||
|  | ||
|
|
||
| ## Validator Node State Diagram | ||
|
|
||
|  |
25 changes: 25 additions & 0 deletions
25
docs/architecture/leader-election/state-diagram-validator.puml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| @startuml state-diagram-validator | ||
|
|
||
| title \n<u>Leader Election State Diagram</u>\n | ||
| footer \nHyperledger Blockchain Integration Framework, 2019 | ||
|
|
||
| [*] --> Started | ||
| Started --> Candidate | ||
| Started: NodeJS process | ||
|
|
||
| Candidate : Attempts to obtain\ngrant on lease\nof Etcd key | ||
| Leader: Sets Etcd key to\n it's own identity | ||
| Follower: Watches Etcd\nkey to determine\nleader's identity | ||
|
|
||
| Candidate -> Follower: lease denied | ||
| Follower -> Candidate: lease TTL expire | ||
| Leader -> Candidate: lease TTL expire | ||
| Candidate -> Leader: lease granted | ||
|
|
||
| Candidate --> Terminated | ||
| Follower --> Terminated | ||
| Leader --> Terminated | ||
|
|
||
| Terminated --> [*] | ||
|
|
||
| @enduml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.