Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Creating three new scenarios #12

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Creating three new scenarios #12

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ab15951
adding split DC only in the inter broker protocol
tomasalmeida May 7, 2024
644aef6
example improvement
tomasalmeida May 8, 2024
7794dd7
on going work
tomasalmeida May 9, 2024
852090c
scenario 9 complete with readme updated
tomasalmeida May 10, 2024
a691f7e
Merge pull request #1 from tomasalmeida/feature/scenario-9
tomasalmeida May 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 31 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ Those scenarios try to emulate a Kafka stretch cluster over 2 datacenters.
- If Zookeeper quorum is rebuilt, there is actually no guarantee that the new quorum have all the latest data:
- this could results in data loss or in messages getting duplicated inside leaders
- it's probably better to rely on hierarchical quorum to avoid those issues
- In case the inter-broker network is lost between data centers, some data is persisted, but the high watermark is not updated until the network is established.

## Scenarios

Expand Down Expand Up @@ -73,4 +74,33 @@ We simulate the following connectivity loss:

All other connections are still up.
All partitions where Kafka-3 is the leader are unavailable.
If we stop Kafka-3, they are still unavailable as unclean leader election is not enabled and Kafka-3 is the only broker in ISR.
If we stop Kafka-3, they are still unavailable as unclean leader election is not enabled and Kafka-3 is the only broker in ISR.

### Scenario 7 - Inter-Broker protocol DC connection loss with partition leader and controller in the same DC

Network setup:
* DC-A: kafka-1, kafka-2, ZK-1, ZK-2
* DC-B: kafka-3, kafka-4, ZK-3

We simulate a DC network split only on the inter-broker protocol level.
ISR is correctly updated, first event is persisted and visible once the network is back.

### Scenario 8 - Inter-Broker protocol DC connection loss with partition leader and controller in different DC

Network setup:
* DC-A: kafka-1, kafka-2, ZK-1, ZK-2
* DC-B: kafka-3, kafka-4, ZK-3

We simulate a DC network split only on the inter-broker protocol level.
ISR is NEVER updated, all events are persisted and visible once the network is back (producer receives exception).

### Scenario 9 - Inter-Broker protocol DC connection loss with leader change

Network setup:
* DC-A: kafka-1, kafka-2, ZK-1, ZK-2
* DC-B: kafka-3, kafka-4, ZK-3

We simulate a DC network split only on the inter-broker protocol level.
Partition leader and controller are in different DC.
DC where the partition leader lives is shut down.
ISR is correctly updated, the event produced in the old leader is discarded (variation of scenario 7 and 8).
2 changes: 2 additions & 0 deletions kafka-streams-1/up
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,5 @@ main() {
}

main

docker-compose down -v
2 changes: 2 additions & 0 deletions scenario-1/up
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,3 +39,5 @@ main() {
}

main

docker-compose down -v
2 changes: 2 additions & 0 deletions scenario-2/up
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,5 @@ main() {
}

main

docker-compose down -v
2 changes: 2 additions & 0 deletions scenario-3/up
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,3 +48,5 @@ main() {
}

main

docker-compose down -v
2 changes: 2 additions & 0 deletions scenario-4/up
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,3 +56,5 @@ main() {
}

main

docker-compose down -v
2 changes: 2 additions & 0 deletions scenario-5/up
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,3 +59,5 @@ main() {
}

main

docker-compose down -v
2 changes: 2 additions & 0 deletions scenario-6/up
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,3 +65,5 @@ main() {
}

main

docker-compose down -v
98 changes: 98 additions & 0 deletions scenario-7/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
version: "3"
services:
zookeeper-1:
build:
context: ../
dockerfile: Dockerfile.zookeeper
hostname: zookeeper-1
environment:
ZOOKEEPER_SERVER_ID: "1"
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_SERVERS: zookeeper-1:2888:3888;zookeeper-2:2888:3888;zookeeper-3:2888:3888
KAFKA_OPTS: -Dzookeeper.4lw.commands.whitelist=*

zookeeper-2:
build:
context: ../
dockerfile: Dockerfile.zookeeper
hostname: zookeeper-2
environment:
ZOOKEEPER_SERVER_ID: "2"
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_SERVERS: zookeeper-1:2888:3888;zookeeper-2:2888:3888;zookeeper-3:2888:3888
KAFKA_OPTS: -Dzookeeper.4lw.commands.whitelist=*

zookeeper-3:
build:
context: ../
dockerfile: Dockerfile.zookeeper
hostname: zookeeper-3
environment:
ZOOKEEPER_SERVER_ID: "3"
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_SERVERS: zookeeper-1:2888:3888;zookeeper-2:2888:3888;zookeeper-3:2888:3888
KAFKA_OPTS: -Dzookeeper.4lw.commands.whitelist=*

kafka-1:
build:
context: ../
dockerfile: Dockerfile.kafka
hostname: kafka-1
depends_on:
- zookeeper-1
- zookeeper-2
- zookeeper-3
environment:
KAFKA_ZOOKEEPER_CONNECT: "zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181"
KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka-1:9092"
KAFKA_OFFSETS_TOPIC_NUM_PARTITIONS: "1"
KAFKA_BROKER_ID: "1"
KAFKA_BROKER_RACK: rack-a

kafka-2:
build:
context: ../
dockerfile: Dockerfile.kafka
hostname: kafka-2
depends_on:
- zookeeper-1
- zookeeper-2
- zookeeper-3
environment:
KAFKA_ZOOKEEPER_CONNECT: "zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181"
KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka-2:9092"
KAFKA_OFFSETS_TOPIC_NUM_PARTITIONS: "1"
KAFKA_BROKER_ID: "2"
KAFKA_BROKER_RACK: rack-a

kafka-3:
build:
context: ../
dockerfile: Dockerfile.kafka
hostname: kafka-3
depends_on:
- zookeeper-1
- zookeeper-2
- zookeeper-3
environment:
KAFKA_ZOOKEEPER_CONNECT: "zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181"
KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka-3:9092"
KAFKA_OFFSETS_TOPIC_NUM_PARTITIONS: "1"
KAFKA_BROKER_ID: "3"
KAFKA_BROKER_RACK: rack-b

kafka-4:
build:
context: ../
dockerfile: Dockerfile.kafka
hostname: kafka-4
depends_on:
- zookeeper-1
- zookeeper-2
- zookeeper-3
environment:
KAFKA_ZOOKEEPER_CONNECT: "zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181"
KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka-4:9092"
KAFKA_OFFSETS_TOPIC_NUM_PARTITIONS: "1"
KAFKA_BROKER_ID: "4"
KAFKA_BROKER_RACK: rack-b
75 changes: 75 additions & 0 deletions scenario-7/up
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
#!/bin/bash

source ../utils/utils.sh

docker-compose down -v
docker-compose up -d

log 'Waiting 30 seconds for containers to start...'
sleep 30

KAFKA_1_IP=$(container_to_ip kafka-1)
KAFKA_2_IP=$(container_to_ip kafka-2)
KAFKA_3_IP=$(container_to_ip kafka-3)
KAFKA_4_IP=$(container_to_ip kafka-4)
interface=eth0

main() {
sleep 20
create_topic zookeeper-1 test 1 4 3
sleep 5

get_state zookeeper-1

send_message_from zookeeper-1 kafka-1 'before network outage'
read_messages kafka-1 1

log 'Controller and partition leader should be in the same DC'

while : ; do
leaderId=`get_state zookeeper-1 | tail -n1 | jq .leader`
controlerId=`get_controller zookeeper-1 | tail -n1 | jq .brokerid`
log "Leader $leaderId and controller $controlerId"
[[ ($leaderId -le 2 && $controlerId -gt 2) || ($leaderId -gt 2 && $controlerId -le 2) ]] || break
force_reelection zookeeper-1
sleep 5
done

get_last_offset kafka-$leaderId

log 'Network outage starts'

# rack A
block_host kafka-1 $KAFKA_3_IP $KAFKA_4_IP
block_host kafka-2 $KAFKA_3_IP $KAFKA_4_IP
# rack B
block_host kafka-3 $KAFKA_1_IP $KAFKA_2_IP
block_host kafka-4 $KAFKA_1_IP $KAFKA_2_IP

log "Let's wait for the timeouts to occur"
sleep 45

send_message_from zookeeper-1 kafka-1 'after network outage via broker 1'
send_message_from zookeeper-1 kafka-3 'after network outage via broker 3'
read_messages_from zookeeper-1 kafka-$leaderId 3
get_state zookeeper-1

get_last_offset kafka-$leaderId

log 'Network outage ends'
remove_partition kafka-1 kafka-2 kafka-3 kafka-4

log 'waiting for the cluster to recover'
sleep 45

send_message_from zookeeper-1 kafka-$leaderId "after network outage ends - via broker $leaderId"
read_messages_from zookeeper-1 kafka-$leaderId 4

get_last_offset kafka-$leaderId

get_state zookeeper-1
}

main

docker-compose down -v
98 changes: 98 additions & 0 deletions scenario-8/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
version: "3"
services:
zookeeper-1:
build:
context: ../
dockerfile: Dockerfile.zookeeper
hostname: zookeeper-1
environment:
ZOOKEEPER_SERVER_ID: "1"
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_SERVERS: zookeeper-1:2888:3888;zookeeper-2:2888:3888;zookeeper-3:2888:3888
KAFKA_OPTS: -Dzookeeper.4lw.commands.whitelist=*

zookeeper-2:
build:
context: ../
dockerfile: Dockerfile.zookeeper
hostname: zookeeper-2
environment:
ZOOKEEPER_SERVER_ID: "2"
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_SERVERS: zookeeper-1:2888:3888;zookeeper-2:2888:3888;zookeeper-3:2888:3888
KAFKA_OPTS: -Dzookeeper.4lw.commands.whitelist=*

zookeeper-3:
build:
context: ../
dockerfile: Dockerfile.zookeeper
hostname: zookeeper-3
environment:
ZOOKEEPER_SERVER_ID: "3"
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_SERVERS: zookeeper-1:2888:3888;zookeeper-2:2888:3888;zookeeper-3:2888:3888
KAFKA_OPTS: -Dzookeeper.4lw.commands.whitelist=*

kafka-1:
build:
context: ../
dockerfile: Dockerfile.kafka
hostname: kafka-1
depends_on:
- zookeeper-1
- zookeeper-2
- zookeeper-3
environment:
KAFKA_ZOOKEEPER_CONNECT: "zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181"
KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka-1:9092"
KAFKA_OFFSETS_TOPIC_NUM_PARTITIONS: "1"
KAFKA_BROKER_ID: "1"
KAFKA_BROKER_RACK: rack-a

kafka-2:
build:
context: ../
dockerfile: Dockerfile.kafka
hostname: kafka-2
depends_on:
- zookeeper-1
- zookeeper-2
- zookeeper-3
environment:
KAFKA_ZOOKEEPER_CONNECT: "zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181"
KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka-2:9092"
KAFKA_OFFSETS_TOPIC_NUM_PARTITIONS: "1"
KAFKA_BROKER_ID: "2"
KAFKA_BROKER_RACK: rack-a

kafka-3:
build:
context: ../
dockerfile: Dockerfile.kafka
hostname: kafka-3
depends_on:
- zookeeper-1
- zookeeper-2
- zookeeper-3
environment:
KAFKA_ZOOKEEPER_CONNECT: "zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181"
KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka-3:9092"
KAFKA_OFFSETS_TOPIC_NUM_PARTITIONS: "1"
KAFKA_BROKER_ID: "3"
KAFKA_BROKER_RACK: rack-b

kafka-4:
build:
context: ../
dockerfile: Dockerfile.kafka
hostname: kafka-4
depends_on:
- zookeeper-1
- zookeeper-2
- zookeeper-3
environment:
KAFKA_ZOOKEEPER_CONNECT: "zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181"
KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka-4:9092"
KAFKA_OFFSETS_TOPIC_NUM_PARTITIONS: "1"
KAFKA_BROKER_ID: "4"
KAFKA_BROKER_RACK: rack-b
Loading