Skip to content

Commit

Permalink
[FAB-2484] Prevent unclean leader election
Browse files Browse the repository at this point in the history
https://jira.hyperledger.org/browse/FAB-2484

This fixes the bug described in:
https://jira.hyperledger.org/browse/FAB-2259

Kafka ships with the `unclean.leader.election.enable` setting set to
`true`. For our use case, this default is wrong as it trades consistency
off for availability.

Concretely, consider this worst-case scenario that goes against every
proper practice:
- Two brokers: B0 and B1
- A replication factor of 2
- `min.insync.replicas` set to 1 (the default)

Assume B1 is the partition leader replica (though this doesn't really
matter).

B1 receives a batch of messages from an ordering service node, and
writes them to disk. B0 does the same, replicating B1.

We shut B0 down.

We send a new batch of messages to the cluster (read: B1); B1 writes
them to disk.

We shut B1 down and then bring B0 up.

If we send a new batch of messages to the cluster (read: B0), it will
write them to disk using _re-using (some/all of) the offsets that B1 had
used for its last batch of messages. This means we are rewriting the
blockchain, which is obviously unacceptable.

This changeset prevents this from happening.

Change-Id: Iaa8f8a9fb3a5e001d4deeebcf2774f2a1026855c
Signed-off-by: Kostas Christidis <kostas@christidis.io>
  • Loading branch information
kchristidis committed Feb 25, 2017
1 parent 2df8355 commit 2c9fd1b
Show file tree
Hide file tree
Showing 5 changed files with 10 additions and 4 deletions.
1 change: 1 addition & 0 deletions bddtests/environments/kafka/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ services:
kafka:
image: hyperledger/fabric-kafka
environment:
KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: "false"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
depends_on:
- zookeeper
1 change: 1 addition & 0 deletions bddtests/environments/orderer-1-kafka-1/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ services:
image: hyperledger/fabric-kafka
environment:
KAFKA_BROKER_ID: 0
KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: "false"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
depends_on:
- zookeeper
9 changes: 6 additions & 3 deletions bddtests/environments/orderer-1-kafka-3/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,25 +22,28 @@ services:
image: hyperledger/fabric-kafka
environment:
KAFKA_BROKER_ID: 0
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_DEFAULT_REPLICATION_FACTOR: 3
KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: "false"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
depends_on:
- zookeeper

kafka1:
image: hyperledger/fabric-kafka
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_DEFAULT_REPLICATION_FACTOR: 3
KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: "false"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
depends_on:
- zookeeper

kafka2:
image: hyperledger/fabric-kafka
environment:
KAFKA_BROKER_ID: 2
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_DEFAULT_REPLICATION_FACTOR: 3
KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: "false"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
depends_on:
- zookeeper
1 change: 1 addition & 0 deletions bddtests/environments/orderer-n-kafka-n/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ services:
kafka:
image: hyperledger/fabric-kafka
environment:
KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: "false"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
depends_on:
- zookeeper
2 changes: 1 addition & 1 deletion orderer/orderer.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ General:

# Log Level: The level at which to log. This accepts logging specifications
# per: fabric/docs/Setup/logging-control.md
LogLevel: info
LogLevel: debug

# Genesis method: The method by which to retrieve/generate the genesis
# block. Available values are "provisional", "file". Provisional utilizes
Expand Down

0 comments on commit 2c9fd1b

Please sign in to comment.