diff --git a/ydb/docs/en/core/reference/kafka-api/constraints.md b/ydb/docs/en/core/reference/kafka-api/constraints.md new file mode 100644 index 000000000000..6d4a5a69d116 --- /dev/null +++ b/ydb/docs/en/core/reference/kafka-api/constraints.md @@ -0,0 +1,17 @@ +# Kafka API constraints + +YDB supports [Apache Kafka protocol](https://kafka.apache.org/protocol.html) version 3.4.0 with the following constraints: + +1. Only authenticated connections are allowed. + +2. Only `SASL/PLAIN` authentication method is supported. + +3. Message compression is not supported. + +4. Only Manual Partition Assignment is supported in read mode, using the [assign method](https://kafka.apache.org/35/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#assign(java.util.Collection)). Consumer group partitions can't be used. + +5. Transactions are not supported. + +6. DDL operations are not supported. Use the [YDB SDK](../ydb-sdk/index.md) or [YDB CLI](../ydb-cli/index.md) to perform them. + +7. Data schema validation not supported. \ No newline at end of file diff --git a/ydb/docs/en/core/reference/kafka-api/examples.md b/ydb/docs/en/core/reference/kafka-api/examples.md new file mode 100644 index 000000000000..1a338b55d79e --- /dev/null +++ b/ydb/docs/en/core/reference/kafka-api/examples.md @@ -0,0 +1,128 @@ +# Kafka API usage examples + +This article provides examples of Kafka API usage to work with [{{ ydb-short-name }} topics](../../concepts/topic.md). + + +Before executing the examples, [create a topic](../ydb-cli/topic-create.md) and [add a consumer](../ydb-cli/topic-consumer-add.md). + +## Examples of working with topics + +The examples use: + + * `ydb:9093` — host name. + * `/Root/Database` — database name. + * `/Root/Database/Topic` — topic name. + * `user@/Root/Database` — username. Includes the username and database name. + * `*****` — user password. + + +## Writing data to a topic + +### Writing via Kafka Java SDK + +This example includes a code snippet for writing data to a topic via [Kafka API](https://kafka.apache.org/documentation/). + + ```java + String HOST = "ydb:9093"; + String TOPIC = "/Root/Database/Topic"; + String USER = "user@/Root/Database"; + String PASS = "*****"; + + Properties props = new Properties(); + props.put("bootstrap.servers", HOST); + props.put("acks", "all"); + + props.put("key.serializer", StringSerializer.class.getName()); + props.put("key.deserializer", StringDeserializer.class.getName()); + props.put("value.serializer", StringSerializer.class.getName()); + props.put("value.deserializer", StringDeserializer.class.getName()); + + props.put("security.protocol", "SASL_SSL"); + props.put("sasl.mechanism", "PLAIN"); + props.put("sasl.jaas.config", PlainLoginModule.class.getName() + " required username=\"" + USER + "\" password=\"" + PASS + "\";"); + + props.put("compression.type", "none"); + + Producer producer = new KafkaProducer<>(props); + producer.send(new ProducerRecord(TOPIC, "msg-key", "msg-body")); + producer.flush(); + producer.close(); + ``` + +### Writing via Logstash + +To configure [Logstash](https://github.com/elastic/logstash), use the following parameters: + + ``` + output { + kafka { + codec => json + topic_id => "/Root/Database/Topic" + bootstrap_servers => "ydb:9093" + compression_type => none + security_protocol => SASL_SSL + sasl_mechanism => PLAIN + sasl_jaas_config => "org.apache.kafka.common.security.plain.PlainLoginModule required username='user@/Root/Database' password='*****';" + } + } + ``` + +### Writing via Fluent Bit + +To configure [Fluent Bit](https://github.com/fluent/fluent-bit), use the following parameters: + + ``` + [OUTPUT] + name kafka + match * + Brokers ydb:9093 + Topics /Root/Database/Topic + rdkafka.client.id Fluent-bit + rdkafka.request.required.acks 1 + rdkafka.log_level 7 + rdkafka.security.protocol SASL_SSL + rdkafka.sasl.mechanism PLAIN + rdkafka.sasl.username user@/Root/Database + rdkafka.sasl.password ***** + ``` + +## Reading data from a topic + +### Reading data from a topic via Kafka Java SDK + +This example includes a code snippet for reading data from a topic via Kafka Java SDK. + +```java + String HOST = "ydb:9093"; + String TOPIC = "/Root/Database/Topic"; + String USER = "user@/Root/Database"; + String PASS = "*****"; + + Properties props = new Properties(); + props.put("bootstrap.servers", HOST); + props.put("auto.offset.reset", "earliest"); // to read from start + props.put("check.crcs", false); + + props.put("key.deserializer", StringDeserializer.class.getName()); + props.put("value.deserializer", StringDeserializer.class.getName()); + + props.put("security.protocol", "SASL_SSL"); + props.put("sasl.mechanism", "PLAIN"); + props.put("sasl.jaas.config", PlainLoginModule.class.getName() + " required username=\"" + USER + "\" password=\"" + PASS + "\";"); + + Consumer consumer = new KafkaConsumer<>(props); + + List partitionInfos = consumer.partitionsFor(TOPIC); + List topicPartitions = new ArrayList<>(); + + for (PartitionInfo partitionInfo : partitionInfos) { + topicPartitions.add(new TopicPartition(partitionInfo.topic(), partitionInfo.partition())); + } + consumer.assign(topicPartitions); + + while (true) { + ConsumerRecords records = consumer.poll(1000); + for (ConsumerRecord record : records) { + System.out.println(record.key() + ":" + record.value()); + } + } \ No newline at end of file diff --git a/ydb/docs/en/core/reference/kafka-api/index.md b/ydb/docs/en/core/reference/kafka-api/index.md new file mode 100644 index 000000000000..64c9da26fc97 --- /dev/null +++ b/ydb/docs/en/core/reference/kafka-api/index.md @@ -0,0 +1,8 @@ +# Kafka API + +{{ ydb-short-name }} supports working with [topics](../../concepts/topic.md) using [the Kafka protocol version 3.4.0](https://kafka.apache.org/34/documentation.html). It allows to integrate {{ ydb-short-name }} with applications originally developed to work with [Apache Kafka](https://kafka.apache.org/). + +The Kafka API documentation contains the following sections: + +* [Usage examples](examples.md) +* [Constraints](constraints.md) diff --git a/ydb/docs/en/core/reference/kafka-api/toc_i.yaml b/ydb/docs/en/core/reference/kafka-api/toc_i.yaml new file mode 100644 index 000000000000..d0ef66df1eab --- /dev/null +++ b/ydb/docs/en/core/reference/kafka-api/toc_i.yaml @@ -0,0 +1,5 @@ +items: + - name: Usage examples + href: examples.md + - name: Constraints + href: constraints.md \ No newline at end of file diff --git a/ydb/docs/en/core/reference/kafka-api/toc_p.yaml b/ydb/docs/en/core/reference/kafka-api/toc_p.yaml new file mode 100644 index 000000000000..bb200f7ddbab --- /dev/null +++ b/ydb/docs/en/core/reference/kafka-api/toc_p.yaml @@ -0,0 +1,4 @@ +items: +- name: Overview + href: index.md +- include: { mode: link, path: toc_i.yaml } \ No newline at end of file diff --git a/ydb/docs/en/core/toc_i.yaml b/ydb/docs/en/core/toc_i.yaml index 2f3bb66a90c5..ddc22688ef02 100644 --- a/ydb/docs/en/core/toc_i.yaml +++ b/ydb/docs/en/core/toc_i.yaml @@ -25,6 +25,7 @@ items: - { name: Compatibility with PostgreSQL, include: { mode: link, path: postgresql/toc_p.yaml } } - { name: Working with the YDB CLI, include: { mode: link, path: reference/ydb-cli/toc_p.yaml } } - { name: Working with the YDB SDK, include: { mode: link, path: reference/ydb-sdk/toc_p.yaml } } +- { name: Working with the Kafka API, include: { mode: link, path: reference/kafka-api/toc_p.yaml } } - { name: Development, include: { mode: link, path: development/toc_p.yaml } } # Footer - { name: Questions and answers, include: { mode: link, path: faq/toc_p.yaml } } diff --git a/ydb/docs/ru/core/reference/kafka-api/constraints.md b/ydb/docs/ru/core/reference/kafka-api/constraints.md index ba1b1680ca0c..3e0fbed72862 100644 --- a/ydb/docs/ru/core/reference/kafka-api/constraints.md +++ b/ydb/docs/ru/core/reference/kafka-api/constraints.md @@ -1,3 +1,5 @@ +# Ограничения Kafka API + Поддержка протокола Kafka версии 3.4.0 осуществляется в ограниченном объеме: 1. Разрешены только аутентифицированные подключения. diff --git a/ydb/docs/ru/core/reference/kafka-api/examples.md b/ydb/docs/ru/core/reference/kafka-api/examples.md index 3c6ea56c9fe7..53770e24f08d 100644 --- a/ydb/docs/ru/core/reference/kafka-api/examples.md +++ b/ydb/docs/ru/core/reference/kafka-api/examples.md @@ -19,7 +19,7 @@ ### Запись через Kafka Java SDK -В этом примере приведен фрагмент кода для записи в топик через Kafka API. +В этом примере приведен фрагмент кода для записи в топик через [Kafka API](https://kafka.apache.org/documentation/). ```java String HOST = "ydb:9093"; diff --git a/ydb/docs/ru/core/reference/kafka-api/index.md b/ydb/docs/ru/core/reference/kafka-api/index.md index b55d7c372681..d265017a8885 100644 --- a/ydb/docs/ru/core/reference/kafka-api/index.md +++ b/ydb/docs/ru/core/reference/kafka-api/index.md @@ -2,6 +2,6 @@ YDB поддерживает работу с топиками по протоколу [Kafka версия 3.4.0](https://kafka.apache.org/34/documentation.html). -[Ограничения использования Kafka API](constraints.md) +[Примеры использования Kafka API](examples.md) -[Примеры использования Kafka API](examples.md) \ No newline at end of file +[Ограничения использования Kafka API](constraints.md) \ No newline at end of file diff --git a/ydb/docs/ru/core/reference/kafka-api/toc_i.yaml b/ydb/docs/ru/core/reference/kafka-api/toc_i.yaml index 978f1c136c2d..7ca3b1f722d6 100644 --- a/ydb/docs/ru/core/reference/kafka-api/toc_i.yaml +++ b/ydb/docs/ru/core/reference/kafka-api/toc_i.yaml @@ -1,5 +1,5 @@ items: - - name: Ограничения - href: constraints.md - name: Примеры использования - href: examples.md \ No newline at end of file + href: examples.md + - name: Ограничения + href: constraints.md \ No newline at end of file