Skip to content

Commit

Permalink
feat: add kafka importer as option (add profiles for kafka and hazelc…
Browse files Browse the repository at this point in the history
…ast, fix schema)
  • Loading branch information
gasymovrv committed Dec 21, 2023
1 parent 5278572 commit 14380fe
Show file tree
Hide file tree
Showing 5 changed files with 40 additions and 11 deletions.
13 changes: 12 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,10 +84,21 @@ Available commit types:

### Zeebe cluster backend for local development

There's a file ```docker/docker-compose.yml``` prepared in this repo, which can be used with recent Docker version to
There's a file [docker-compose.yml](docker/docker-compose.yml) prepared in this repo, which can be used with recent Docker version to
provide a backend. You just need to alter this file and comment out the ```monitor``` service. What remains is
the ```zeebe``` service, which then can be started via ```docker compose up``` command.

There are several profiles in the file for running services in the prepared configuration.
They can be specified in a file [.env](docker/.env), by passing multiple --profile flags or a comma-separated list for the COMPOSE_PROFILES environment variable:
* ```docker compose --profile hazelcast --profile hazelcast_in_memory up```
* ```COMPOSE_PROFILES=hazelcast,hazelcast_in_memory docker compose up```

Now there are only these configuration presets:
* ```COMPOSE_PROFILES=hazelcast,hazelcast_in_memory```
* ```COMPOSE_PROFILES=kafka,kafka_in_memory```
* ```COMPOSE_PROFILES=hazelcast,hazelcast_postgres,postgres```
* ```COMPOSE_PROFILES=hazelcast,hazelcast_mysql,mysql```

With such a backend running, you can simply start debugging the ```ZeebeSimpleMonitorApp``` class in your IDE of choice.

### Reading 'text' fields in PostgreSQL
Expand Down
21 changes: 15 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,16 +67,25 @@ docker run --network="host" ghcr.io/camunda-community-hub/zeebe-simple-monitor:2
```

For a local setup, the repository contains a [docker-compose file](docker/docker-compose.yml). It starts a Zeebe broker with the Hazelcast/Kafka exporter and the application.

There are several docker-compose profiles, setting by a file [.env](docker/.env).
Existing presets:
* ```COMPOSE_PROFILES=hazelcast,hazelcast_in_memory``` (by default)
* ```COMPOSE_PROFILES=kafka,kafka_in_memory```
* ```COMPOSE_PROFILES=hazelcast,hazelcast_postgres,postgres```
* ```COMPOSE_PROFILES=hazelcast,hazelcast_mysql,mysql```

The commands to build and run:
```
mvn clean install -DskipTests
cd docker
docker-compose --profile in-memory up
docker-compose up
```

Go to http://localhost:8082

To use PostgreSQL instead of the in-memory database, use the profile `postgres`.
To change the database see "[Change the Database](#change-the-database)"

To change Zeebe importer see "[Change the default Zeebe importer to Kafka](#change-the-default-zeebe-importer-to-kafka)"

```
docker-compose --profile postgres up
Expand Down Expand Up @@ -214,7 +223,7 @@ For example, using PostgreSQL:

* the PostgreSQL database driver is already bundled

See the [docker-compose file](docker/docker-compose.yml) (profile: `postgres`) for a sample configuration with PostgreSQL.
See the [docker-compose file](docker/docker-compose.yml) for a sample configuration with PostgreSQL. Profiles presets: `hazelcast,hazelcast_postgres,postgres`

The configuration for using MySql is similar but with an additional setting for the Hibernate naming strategy:

Expand All @@ -229,7 +238,7 @@ The configuration for using MySql is similar but with an additional setting for

* the MySql database driver is already bundled

See the [docker-compose file](docker/docker-compose.yml) (profile: `mysql`) for a sample configuration with MySql.
See the [docker-compose file](docker/docker-compose.yml) for a sample configuration with MySql. Profiles presets: `hazelcast,hazelcast_mysql,mysql`

#### Change the default Zeebe importer to Kafka

Expand All @@ -240,7 +249,7 @@ See the [docker-compose file](docker/docker-compose.yml) (profile: `mysql`) for
* `spring.kafka.custom.concurrency` (default: `3`) is the number of threads for the Kafka listener that will import events from Zeebe
* `spring.kafka.custom.retry.intervalMs` (default: `30000`) and `spring.kafka.custom.retry.max-attempts` (default: `3`) are the retry configurations for a retryable exception in the listener

Refer to the [docker-compose file](docker/docker-compose.yml) (service: `simple-monitor-in-memory-kafka`) for a sample configuration with the Kafka importer.
Refer to the [docker-compose file](docker/docker-compose.yml) for a sample configuration with the Kafka importer. Profiles presets: `kafka,kafka_in_memory`

## Code of Conduct

Expand Down
1 change: 1 addition & 0 deletions docker/.env
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
COMPOSE_PROFILES=hazelcast,hazelcast_in_memory
16 changes: 12 additions & 4 deletions docker/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ services:
- zeebe_network
volumes:
- ./hazelcast/application.yaml:/usr/local/zeebe/config/application.yaml
profiles:
- hazelcast

zeebe-kafka:
container_name: zeebe-broker-kafka
Expand All @@ -38,6 +40,8 @@ services:
volumes:
- ./kafka/exporter.yml:/usr/local/zeebe/config/exporter.yml
- ./kafka/exporter/zeebe-kafka-exporter-3.1.1-jar-with-dependencies.jar:/usr/local/zeebe/lib/zeebe-kafka-exporter.jar
profiles:
- kafka

zookeeper:
image: docker.io/bitnami/zookeeper:3.8
Expand All @@ -47,6 +51,8 @@ services:
- ALLOW_ANONYMOUS_LOGIN=yes
networks:
- zeebe_network
profiles:
- kafka

kafka:
image: docker.io/bitnami/kafka:3.4
Expand All @@ -64,6 +70,8 @@ services:
- zookeeper
networks:
- zeebe_network
profiles:
- kafka

simple-monitor-in-memory:
container_name: zeebe-simple-monitor-in-memory
Expand All @@ -79,7 +87,7 @@ services:
networks:
- zeebe_network
profiles:
- in-memory
- hazelcast_in_memory

simple-monitor-in-memory-kafka:
container_name: zeebe-simple-monitor-in-memory-kafka
Expand All @@ -96,7 +104,7 @@ services:
networks:
- zeebe_network
profiles:
- in-memory
- kafka_in_memory

simple-monitor-postgres:
container_name: zeebe-simple-monitor-postgres
Expand All @@ -118,7 +126,7 @@ services:
networks:
- zeebe_network
profiles:
- postgres
- hazelcast_postgres

postgres-zeebe-simple-monitor:
image: postgres:16.1
Expand Down Expand Up @@ -153,7 +161,7 @@ services:
networks:
- zeebe_network
profiles:
- mysql
- hazelcast_mysql

mysql-zeebe-simple-monitor:
image: mysql:8
Expand Down
Binary file modified docs/how-it-works.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 14380fe

Please sign in to comment.