Implement Dev Services for Kafka

docs/src/main/asciidoc/kafka-dev-services.adoc

@@ -0,0 +1,40 @@
+////
+This guide is maintained in the main Quarkus repository
+and pull requests should be submitted there:
+https://github.com/quarkusio/quarkus/tree/main/docs/src/main/asciidoc
+////
+= Dev Services for Kafka
+
+include::./attributes.adoc[]
+
+Dev Services for Kafka automatically starts a Kafka broker in dev mode and when running tests.
+So, you don't have to start a broker manually.
+The application is configured automatically.
+
+IMPORTANT: Because starting a Kafka broker can be long, Dev Services for Kafka uses https://vectorized.io/redpanda[Red Panda], a Kafka compatible broker which starts in ~1 second.
+
+== Enabling / Disabling Dev Services for Kafka
+
+Dev Services for Kafka is automatically enabled unless:
+
+- `quarkus.kafka.devservices.enabled` is set to `false`
+- the `kafka.bootstrap.servers` is configured
+- all the Reactive Messaging Kafka channels have the `bootstrap.servers` attribute set
+
+Dev Services for Kafka relies on Docker to start the broker.
+If your environment does not support Docker, you will need to start the broker manually, or connect to an already running broker.
+You can configure the broker address using `kafka.bootstrap.servers`.
+
+== Setting the port
+
+By default, Dev Services for Kafka picks a random port and configure the application.
+You can set the port by configuring the `quarkus.kafka.devservices.port` property.
+
+Note that the Kafka advertised address is automatically configured with the chosen port.
+
+== Configuring the image
+
+By default, Dev Services for Kafka uses: `vectorized/redpanda:latest`.
+You can select any version from https://hub.docker.com/r/vectorized/redpanda.
+
+IMPORTANT: Dev Services for Kafka only support Red Panda.

docs/src/main/asciidoc/kafka.adoc

@@ -74,48 +74,12 @@ This will add the following to your `pom.xml`:
 </dependency>
 ----
 
-== Starting Kafka
-
-Then, we need a Kafka cluster.
-You can follow the instructions from the https://kafka.apache.org/quickstart[Apache Kafka web site] or create a `docker-compose.yaml` file with the following content:
-
-[source, yaml]
-----
-version: '2'
-
-services:
-
-  zookeeper:
-    image: strimzi/kafka:0.19.0-kafka-2.5.0
-    command: [
-      "sh", "-c",
-      "bin/zookeeper-server-start.sh config/zookeeper.properties"
-    ]
-    ports:
-      - "2181:2181"
-    environment:
-      LOG_DIR: /tmp/logs
-
-  kafka:
-    image: strimzi/kafka:0.19.0-kafka-2.5.0
-    command: [
-      "sh", "-c",
-      "bin/kafka-server-start.sh config/server.properties --override listeners=$${KAFKA_LISTENERS} --override advertised.listeners=$${KAFKA_ADVERTISED_LISTENERS} --override zookeeper.connect=$${KAFKA_ZOOKEEPER_CONNECT}"
-    ]
-    depends_on:
-      - zookeeper
-    ports:
-      - "9092:9092"
-    environment:
-      LOG_DIR: "/tmp/logs"
-      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
-      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
-      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
-----
-
-Once created, run `docker-compose up`.
-
-NOTE: This is a development cluster, do not use in production.
+[TIP]
+.Dev Services
+====
+No need to start a Kafka broker when using the dev mode or for tests. Quarkus starts a broker for you automatically.
+See xref:kafka-dev-services.adoc[Dev Services for Kafka] for details.
+====
 
 == The price generator
 
@@ -256,9 +220,6 @@ The `channel-name` segment must match the value set in the `@Incoming` and `@Out
 
 [source,properties]
 ----
-# Configure the SmallRye Kafka connector
-kafka.bootstrap.servers=localhost:9092
-
 # Configure the Kafka sink (we write to it)
 mp.messaging.outgoing.generated-price.connector=smallrye-kafka
 mp.messaging.outgoing.generated-price.topic=prices
@@ -286,6 +247,10 @@ So, if you are running in an environment only routing traffic to containers that
 More details about health reporting is given in <<kafka-health-check>>.
 ====
 
+The previous configuration does not set the Kafka _bootstrap.servers_.
+Quarkus starts a Kafka broker automatically and configures the application.
+See xref:kafka-dev-services.adoc[Dev Services for Kafka] for more details.
+
 == The HTML page
 
 Final touch, the HTML page reading the converted prices using SSE.
@@ -328,8 +293,7 @@ Nothing spectacular here. On each received price, it updates the page.
 
 == Get it running
 
-If you followed the instructions, you should have Kafka running.
-Then, you just need to run the application using:
+You just need to run the application using:
 
 [source,bash]
 ----
@@ -338,15 +302,66 @@ Then, you just need to run the application using:
 
 Open `http://localhost:8080/prices.html` in your browser.
 
-NOTE: If you started the Kafka broker with docker compose, stop it using `CTRL+C` followed by `docker-compose down`.
+== Running in JVM or Native mode
+
+When not running in dev or test mode, you will need to start your Kafka broker.
+You can follow the instructions from the https://kafka.apache.org/quickstart[Apache Kafka web site] or create a `docker-compose.yaml` file with the following content:
+
+[source, yaml]
+----
+version: '2'
+
+services:
+
+  zookeeper:
+    image: strimzi/kafka:0.19.0-kafka-2.5.0
+    command: [
+      "sh", "-c",
+      "bin/zookeeper-server-start.sh config/zookeeper.properties"
+    ]
+    ports:
+      - "2181:2181"
+    environment:
+      LOG_DIR: /tmp/logs
+
+  kafka:
+    image: strimzi/kafka:0.19.0-kafka-2.5.0
+    command: [
+      "sh", "-c",
+      "bin/kafka-server-start.sh config/server.properties --override listeners=$${KAFKA_LISTENERS} --override advertised.listeners=$${KAFKA_ADVERTISED_LISTENERS} --override zookeeper.connect=$${KAFKA_ZOOKEEPER_CONNECT}"
+    ]
+    depends_on:
+      - zookeeper
+    ports:
+      - "9092:9092"
+    environment:
+      LOG_DIR: "/tmp/logs"
+      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
+      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
+      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
+----
+
+Once created, run `docker-compose up`.
+
+NOTE: This is a development cluster, do not use in production.
+
+You can build and run the application in JVM mode with:
+
+[source, bash]
+----
+./mvnw package
+java -jar target/quarkus-app/quarkus-run.jar
+----
 
-== Running Native
+NOTE: By default, the application tries to connect to a Kafka broker listening at `localhost:9092`.
+You can configure the bootstrap server using: `java -Dkafka.bootstrap.servers=... -jar target/quarkus-app/quarkus-run.jar`
 
-You can build the native executable with:
+You can build and run the native executable with:
 
 [source,bash]
 ----
 ./mvnw package -Pnative
+./target/kafka-quickstart-1.0.0-SNAPSHOT-runner -Dkafka.bootstrap.servers=localhost:9092
 ----
 
 == Imperative usage

extensions/kafka-client/deployment/pom.xml

@@ -39,6 +39,10 @@
             <groupId>io.quarkus</groupId>
             <artifactId>quarkus-caffeine-deployment</artifactId>
         </dependency>
+        <dependency>
+            <groupId>org.testcontainers</groupId>
+            <artifactId>testcontainers</artifactId>
+        </dependency>
     </dependencies>
 
     <build>

extensions/kafka-client/deployment/src/main/java/io/quarkus/kafka/client/deployment/DevServicesKafkaBrokerBuildItem.java

@@ -0,0 +1,17 @@
+package io.quarkus.kafka.client.deployment;
+
+import io.quarkus.builder.item.SimpleBuildItem;
+
+public final class DevServicesKafkaBrokerBuildItem extends SimpleBuildItem {
+
+    final String bootstrapServers;
+
+    public DevServicesKafkaBrokerBuildItem(String bs) {
+        this.bootstrapServers = bs;
+    }
+
+    public String getBootstrapServers() {
+        return bootstrapServers;
+    }
+
+}