| Confluent Container Version | Kafka Equivalent |
|---|---|
confluentinc/cp-kafka:7.7.0 |
org.apache.kafka/kafka:3.7.0 |
- Introduction
- Prequisites
- Run Kpow Community Edition (optional)
- Run a simple Kafka cluster (no authentication)
- Run a SASL Kafka cluster (with authentication)
- License
This repository contains docker compose configuration to run a local Kafka cluster with Docker Compose.
We use similar configuration for local development of our Kafka UI and API product, Kpow for Apache Kafka.
Two types of Kafka Cluster are supported, simple (no authentication) and SASL authenticated.
See kpow-local for a more complex local configuration consisting of Kpow, Kafka, Schema, Connect, and ksqlDB.
The local cluster runs with Docker Compose, so you will need to install Docker.
Once Docker is installed, clone this repository and run the following commands from the base path.
git clone git@github.com:factorhouse/kafka-local.git
cd kafka-local
The community edition of Kpow for Apache Kafka is free to use by individuals and organisations.
You can check it out if you like, but the configuration in this repository doesn't require Kpow in any way.
Start a local Kafka cluster with the configuration in this repository then:
- Get a free Kpow Community license
- Enter the license details into resoources/kpow/no-auth.env or resoources/kpow/sasl-auth.env
- Start Kpow Community Edition:
Start Kpow Community Edition with No Auth Kafka Cluster
docker run --network=kafka-local_default -p 3000:3000 -m2G --env-file ./resources/kpow/no-auth.env factorhouse/kpow-ce:latest
Start Kpow Community Edition with SASL Auth Kafka Cluster
docker run --network=kafka-local_default -p 3000:3000 -m2G --env-file ./resources/kpow/sasl-auth.env factorhouse/kpow-ce:latest
- Navigate to http://localhost:3000 (the UI might look empty until you start creating topics and writing data)
This command starts a Kafka Cluster that does not require clients to authenticate.
docker compose -f docker-compose-no-auth.yml up[+] Running 5/5
✔ Network kafka-local_default Created0.0s
✔ Container zookeeper Created0.0s
✔ Container kafka-local-kafka-3-1 Created0.0s
✔ Container kafka-local-kafka-1-1 Created0.0s
✔ Container kafka-local-kafka-2-1 Created0.0s
Attaching to kafka-1-1, kafka-2-1, kafka-3-1, zookeeper
zookeeper | ===> User
zookeeper | uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
zookeeper | ===> Configuring ...
First, hit ctrl-c in the terminal running the Docker Compose process.
^C
Gracefully stopping... (press Ctrl+C again to force)
[+] Stopping 4/4
✔ Container kafka-local-kafka-1-1 Stopped5.8s
✔ Container kafka-local-kafka-2-1 Stopped0.7s
✔ Container kafka-local-kafka-3-1 Stopped0.7s
✔ Container zookeeper Stopped0.5s
canceledThen stop/clear the Docker Compose resources
docker compose -f docker-compose-no-auth.yml down
[+] Running 5/0
✔ Container kafka-local-kafka-1-1 Removed 0.0s
✔ Container kafka-local-kafka-2-1 Removed 0.0s
✔ Container kafka-local-kafka-3-1 Removed 0.0s
✔ Container zookeeper Removed 0.0s
✔ Network kafka-local_default Removed
To access this cluster you can:
- Connect to the bootstrap on localhost / 127.0.0.1 (most likely non-docker applications)
- Connect to the bootstrap on the Docker defined hosts (kakfa-1, kafka-2, kafka-3)
- Connect to the bootstrap using
host.docker.internalwhich is similar to (1)
Applications that are external to Docker can access the Kafka cluster via the Localhost bootstrap.
bootstrap: 127.0.0.1:9092,127.0.0.1:9093,127.0.0.1:9094
Containerized applications can connect to the Kafka cluster via the Docker Host bootstrap.
These docker hosts (kakfa-1, kafka-2, kafka-3) are defined within the comoose.yml.
When starting your Docker container, specify that it should share the kafka-local_default network.
docker run --network=kafka-local_default ...
Then connect to the hosts that are running on that network
bootstrap: kafka-1:19092,kafka-2:19093,kafka-3:19094
This is a good trick for running a docker container that connects back to a port open on the host machine.
host.docker.internal effective routes back to localhost.
bootstrap: host.docker.internal:9092,host.docker.internal:9093,host.docker.internal:9094
Use the docker-compose-sasl-auth.yml configuration to run a SASL authenticated cluster:
docker compose -f docker-compose-sasl-auth.yml up
Bootstrap configuration is the same as the simple cluster however clients must authenticate to connect.
Authentication configuration is specified in resources/docker/kafka_jaas.conf.
To connect a client to this client use the following connection settings:
security.protocol: SASL_PLAINTEXT
sasl.mechanism: PLAIN
sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="client" password="client-secret";
This repository is released under the Apache 2.0 License.
Copyright © Factor House.