Playground for Kafka/Confluent Docker experimentations...
ℹ️ How to run
- 🔗 Kafka Connectors
- ☁️ Confluent Cloud
- 🔄 Confluent Replicator and Mirror Maker 2
- 🔐 Environments
- Confluent Commercial
- CP-Ansible Playground
- 👾 Other Playgrounds
- 📚 Useful Resources
Quick start examples from Confluent docs but in Docker version for ease of use.
| Connector | Product Category | Latest Version (*) | Type | Release Date |  Travis |
|---|---|---|---|---|---|
 ActiveMQ Sink |
Message Queue | 1.1.6 | Confluent Subscription | 2020-08-21 | ✅ |
| ActiveMQ Source |
Message Queue | 5.5.2 | Confluent Subscription | 2020-09-30 | ✅ |
 Amazon CloudWatch Logs Source |
Analytics | 1.0.4 | Confluent Subscription | 2020-06-03 | ✅ |
| Amazon CloudWatch Metrics Sink |
Analytics | 1.1.2 | Confluent Subscription | 2020-06-08 | ✅ |
 Amazon DynamoDB Sink |
Database | 1.1.1 | Confluent Subscription | 2020-08-20 | ✅ |
 Amazon Kinesis Source |
Message Queue | 1.3.0 | Confluent Subscription | 2020-08-31 | ✅ |
 Amazon Redshift Sink |
Data Warehouse | 1.0.2 | Confluent Subscription | 2020-04-09 | ✅ |
| Amazon Redshift Source (using JDBC) |
Data Warehouse | 5.5.2 | Confluent Community License | 2020-09-30 | ✅ |
 Amazon S3 Sink |
Datastore | 5.5.2 | Confluent Community License | 2020-09-30 | ✅ |
| Amazon S3 Source |
Datastore | 1.3.2 | Confluent Subscription | 2020-06-26 | ✅ |
 Amazon SQS Source |
Message Queue | 1.0.6 | Confluent Subscription | 2020-09-23 | ✅ |
 AMPS Source |
Message Queue | 1.0.0-preview | Confluent Subscription | 2020-07-08 | ✅ |
 Apache Kudu Source |
Database | 1.0.1 | Confluent Subscription | 2020-01-07 | ✅ |
| Apache Kudu Sink |
Database | 1.0.1 | Confluent Subscription | 2020-01-07 | ✅ |
 AWS Lambda Sink |
SaaS Apps | 1.0.4 | Confluent Subscription | 2020-10-01 | ✅ |
 Azure Blob Storage Sink |
Datastore | 1.4.0 | Confluent Subscription | 2020-07-24 | ✅ |
| Azure Blob Storage Source |
Datastore | 1.3.2 | Confluent Subscription | 2020-06-26 | ✅ |
 Azure Data Lake Storage Gen1 Sink |
Datastore | 1.4.0 | Confluent Subscription | 2020-07-24 | ✅ |
| Azure Data Lake Storage Gen2 Sink |
Datastore | 1.4.0 | Confluent Subscription | 2020-07-24 | ✅ |
 Azure Event Hubs Source |
Message Queue | 1.0.3 | Confluent Subscription | 2020-03-28 | ✅ |
 Azure Search Sink |
Analytics | 1.0.1 | Confluent Subscription | 2020-01-07 | ✅ |
 Azure Service Bus Source |
Message Queue | 1.1.0 | Confluent Subscription | 2020-03-13 | ✅ |
 Azure SQL Data Warehouse Sink |
Data Warehouse | 1.0.3 | Confluent Subscription | 2020-04-24 | ❌ |
 Cassandra Sink |
Database | 1.2.2 | Confluent Subscription | 2020-06-19 | ✅ |
 Couchbase Sink |
Database | 3.4.8 | Open Source (Couchbase) | ✅ | |
| Couchbase Source |
Database | 3.4.8 | Open Source (Couchbase) | ✅ | |
 Debezium CDC Microsoft SQL Server Source |
CDC | 1.2.2 | Open Source (Debezium Community) | ✅ | |
 Debezium CDC MySQL Source |
CDC | 1.2.2 | Open Source (Debezium Community) | ✅ | |
 Debezium CDC PostgreSQL Source |
CDC | 1.2.2 | Open Source (Debezium Community) | ✅ | |
 Debezium CDC MongoDB Source |
CDC | 1.2.2 | Open Source (Debezium Community) | ✅ | |
 Data Diode Sink |
Logs | 1.1.1 | Confluent Subscription | 2019-10-18 | ✅ |
| Data Diode Source |
Logs | 1.1.1 | Confluent Subscription | 2019-10-18 | ✅ |
 Datadog Metrics Sink |
Analytics | 1.1.2 | Confluent Subscription | 2020-06-08 | ❌ |
 ElasticSearch Sink |
Analytics | 10.0.0 | Confluent Community License | 2020-09-24 | ✅ |
 FTPS Sink |
Datastore | 1.0.3-preview | Confluent Subscription | 2020-10-01 | ✅ |
| FTPS Source |
Datastore | 1.0.3-preview | Confluent Subscription | 2020-10-01 | ✅ |
 Gemfire Sink |
Database | 1.0.1 | Confluent Subscription | 2019-11-19 | ✅ |
 Github Source |
SaaS Apps | 1.0.1 | Confluent Subscription | 2020-06-05 | ✅ |
 Google BigQuery Sink |
Data Warehouse | 1.6.1 | Open Source (WePay) | ✅ | |
 Google Cloud Functions Sink |
SaaS Apps | 1.1.1 | Confluent Subscription | 2020-09-22 | ✅ |
 Google Cloud Pub/Sub Source |
Message Queue | 1.0.2 | Confluent Subscription | 2020-01-07 | ✅ |
 Google Cloud Spanner Sink |
Database | 1.0.2 | Confluent Subscription | 2019-10-30 | ❌ |
 Google Cloud Storage Sink |
Datastore | 5.5.3 | Confluent Subscription | 2020-10-01 | ✅ |
| Google Cloud Storage Source |
Datastore | 1.3.2 | Confluent Subscription | 2020-06-26 | ✅ |
 Google Firebase Realtime Database Sink |
Database | 1.1.1 | Confluent Subscription | 2020-01-07 | ❌ |
| Google Firebase Realtime Database Source |
Database | 1.1.1 | Confluent Subscription | 2020-01-07 | ❌ |
 HBase Sink |
Database | 1.0.5 | Confluent Subscription | 2020-07-20 | ✅ |
 HDFS 2 Source |
Datastore | 1.3.2 | Confluent Subscription | 2020-06-26 | ✅ |
| HDFS 3 Source |
Datastore | 1.3.2 | Confluent Subscription | 2020-06-26 | ✅ |
| HDFS 2 Sink |
Datastore | 5.5.2 | Confluent Community License | 2020-09-30 | ✅ |
| HDFS 3 Sink |
Datastore | 1.0.5 | Confluent Subscription | 2020-03-04 | ✅ |
 HTTP Sink |
SaaS Apps | 1.0.16 | Confluent Subscription | 2020-08-03 | ✅ |
 IBM MQ Sink |
Message Queue | 1.3.0 | Confluent Subscription | 2020-09-15 | ✅ |
| IBM MQ Source |
Message Queue | 5.5.2 | Confluent Subscription | 2020-09-30 | ✅ |
 InfluxDB Sink |
Database | 1.2.1 | Confluent Subscription | 2020-09-30 | ✅ |
| InfluxDB Source |
Database | 1.2.1 | Confluent Subscription | 2020-09-30 | ✅ |
 JDBC Hive Sink |
Database | 5.5.2 | Confluent Community License | 2020-09-30 | ✅ |
| JDBC MySQL Sink |
Database | 5.5.2 | Confluent Community License | 2020-09-30 | ✅ |
 JDBC Oracle 11 Sink |
Database | 5.5.2 | Confluent Community License | 2020-09-30 | ✅ |
 JDBC Oracle 12 Sink |
Database | 5.5.2 | Confluent Community License | 2020-09-30 | ✅ |
| JDBC PostGreSQL Sink |
Database | 5.5.2 | Confluent Community License | 2020-09-30 | ✅ |
| JDBC Microsoft SQL Server Sink |
Database | 5.5.2 | Confluent Community License | 2020-09-30 | ✅ |
 JDBC Vertica Sink |
Database | 5.5.2 | Confluent Community License | 2020-09-30 | ✅ |
| JDBC MySQL Source |
Database | 5.5.2 | Confluent Community License | 2020-09-30 | ✅ |
| JDBC Oracle 11 Source |
Database | 5.5.2 | Confluent Community License | 2020-09-30 | ✅ |
| JDBC Oracle 12 Source |
Database | 5.5.2 | Confluent Community License | 2020-09-30 | ✅ |
| JDBC PostGreSQL Source |
Database | 5.5.2 | Confluent Community License | 2020-09-30 | ✅ |
| JDBC Microsoft SQL Server Source |
Database | 5.5.2 | Confluent Community License | 2020-09-30 | ✅ |
 JIRA Source |
SaaS Apps | 1.0.0-preview | Confluent Subscription | 2020-03-30 | ✅ |
| JMS ActiveMQ Sink |
Message Queue | 1.3.0 | Confluent Subscription | 2020-09-15 | ✅ |
 JMS Solace Sink |
Message Queue | 1.3.0 | Confluent Subscription | 2020-09-15 | ✅ |
 JMS TIBCO EMS Sink |
Message Queue | 1.3.0 | Confluent Subscription | 2020-09-15 | ✅ |
| JMS TIBCO EMS Source |
Message Queue | 5.5.2 | Confluent Subscription | 2020-09-30 | ✅ |
 Mapr Sink |
Datastore | 1.1.1 | Confluent Subscription | 2020-02-10 | ❌ |
 Minio Sink |
Datastore | 5.5.2 | Confluent Community License | 2020-09-30 | ✅ |
| MongoDB Sink |
Database | 1.2.0 | Open Source (MongoDB) | 2020-07-08 | ✅ |
| MongoDB Source |
Database | 1.2.0 | Open Source (MongoDB) | 2020-07-08 | ✅ |
 MQTT Sink |
IoT | 1.3.0 | Confluent Subscription | 2020-08-28 | ✅ |
| MQTT Source |
IoT | 1.3.0 | Confluent Subscription | 2020-08-28 | ✅ |
 Neo4j Sink |
Database | 1.0.9 | Open Source (Neo4j, Inc.) | 2020-09-02 | ✅ |
 OmniSci Sink |
Database | 1.0.2 | Confluent Subscription | 2019-08-20 | ✅ |
 PagerDuty Sink |
SaaS Apps | 1.0.1 | Confluent Subscription | 2020-07-20 | ✅ |
 Prometheus Sink |
Analytics | 1.1.2-preview | Confluent Subscription | 2020-06-08 | ✅ |
 RabbitMQ Sink |
Message Queue | 1.3.1-preview | Confluent Subscription | 2020-09-22 | ✅ |
| RabbitMQ Source |
Message Queue | 1.3.1 | Confluent Subscription | 2020-09-22 | ✅ |
 Redis Sink |
Database | 0.0.2.11 | Open Source (Jeremy Custenborder) | 2020-01-22 | ✅ |
 SalesForce Bulk API Sink |
SaaS Apps | 1.7.0 | Confluent Subscription | 2020-09-18 | ✅ |
| SalesForce Bulk API Source |
SaaS Apps | 1.7.0 | Confluent Subscription | 2020-09-18 | ✅ |
| SalesForce CDC Source |
SaaS Apps | 1.7.0 | Confluent Subscription | 2020-09-18 | ✅ |
| SalesForce Platform Events Sink |
SaaS Apps | 1.7.0 | Confluent Subscription | 2020-09-18 | ✅ |
| SalesForce Platform Events Source |
SaaS Apps | 1.7.0 | Confluent Subscription | 2020-09-18 | ✅ |
| SalesForce PushTopics Source |
SaaS Apps | 1.7.0 | Confluent Subscription | 2020-09-18 | ✅ |
| SalesForce SObject Sink |
SaaS Apps | 1.7.0 | Confluent Subscription | 2020-09-18 | ✅ |
 ServiceNow Sink |
SaaS Apps | 2.0.1 | Confluent Subscription | 2020-07-28 | ❌ |
| ServiceNow Source |
SaaS Apps | 2.0.1 | Confluent Subscription | 2020-07-28 | ❌ |
 SFTP Sink |
Datastore | 2.1.1 | Confluent Subscription | 2020-09-02 | ✅ |
| SFTP Sink |
Datastore | 2.1.1 | Confluent Subscription | 2020-09-02 | ✅ |
 SNMP Trap Source |
IoT | 1.1.2 | Confluent Subscription | 2020-04-09 | ✅ |
 Snowflake Sink |
Data Warehouse | 1.4.3 | Open Source (Snowflake, Inc.) | 2020-08-05 | ❌ |
| Solace Sink |
Message Queue | 1.3.0 | Confluent Subscription | 2020-09-15 | ✅ |
| Solace Source |
Message Queue | 1.2.0 | Confluent Subscription | 2020-08-26 | ✅ |
 Splunk Sink |
Analytics | 2.0 | Open Source (Splunk) | ✅ | |
| Splunk Source |
Analytics | 1.0.2 | Confluent Subscription | 2020-01-08 | ✅ |
 Spool Dir Source |
Datastore | 2.0.46 | Open Source (Jeremy Custenborder) | 2020-08-26 | ✅ |
 Syslog Source |
Logs | 1.3.1 | Confluent Subscription | 2020-07-22 | ✅ |
| TIBCO EMS Sink |
Message Queue | 1.3.0 | Confluent Subscription | 2020-09-15 | ✅ |
| TIBCO EMS Source |
Message Queue | 1.2.0 | Confluent Subscription | 2020-08-26 | ✅ |
| Vertica Sink |
Database | 1.2.2 | Confluent Subscription | 2020-09-07 | ✅ |
 Zendesk Source |
SaaS Apps | 1.0.1 | Confluent Subscription | 2020-06-04 | ✅ |
* You can change default connector version by setting CONNECTOR_TAG environment variable before starting a test, get more details here
- How to connect your components to Confluent Cloud
- How to monitor your Confluent Cloud cluster
- How to restrict access
- etc...
ServiceNow source- ServiceNow sink
MongoDB source
Firebase
- Using cp-ansible with Confluent Cloud
- Demo using dabz/ccloudexporter in order to pull Metrics API data from Confluent Cloud cluster and export it to Prometheus (Grafana dashboard is also available)
.NET client (producer/consumer)
Go client (producer/consumer)kafka-admin Managing topics and ACLs using matt-mangia/kafka-admin
Confluent Replicator OnPrem to cloud and Cloud to Cloud examples
Multi-Cluster Schema Registry with hybrid configuration (onprem/confluent cloud)
- Confluent REST Proxy Security Plugin with Principal Propagation
- Migrate Schemas to Confluent Cloud using Confluent Replicator
Using Multi-Data-Center setup with US 🇺🇸 and EUROPE 🇪🇺 clusters.
Using Confluent Replicator as connector
- Using PLAINTEXT
- Using SASL_PLAIN
- Using Kerberos
- 👾 Using Confluent Replicator as executable
- Using PLAINTEXT
- Using SASL_PLAIN
- Using Kerberos
Using Mirror Maker 2
- Using PLAINTEXT
Single cluster:
- PLAINTEXT: no security
- SASL_PLAIN: no SSL encryption, SASL/PLAIN authentication
- SASL/SCRAM no SSL encryption, SASL/SCRAM-SHA-256 authentication
- SASL_SSL: SSL encryption, SASL/PLAIN authentication
- 2WAY_SSL: SSL encryption, SSL authentication
- Kerberos: no SSL encryption, Kerberos GSSAPI authentication
- SSL_Kerberos SSL encryption, Kerberos GSSAPI authentication
- LDAP Authorizer with SASL/PLAIN no SSL encryption, SASL/PLAIN authentication, LDAP Authorizer for ACL authorization
- RBAC with SASL/PLAIN RBAC with no SSL encryption, SASL/PLAIN authentication
Multi-Data-Center setup:
- PLAINTEXT: no security
- SASL_PLAIN: no SSL encryption, SASL/PLAIN authentication
- Kerberos: no SSL encryption, Kerberos GSSAPI authentication
- Control Center
- Tiered Storage
- Tiered storage with AWS S3
- Tiered storage with Minio (unsupported)
- Confluent Rebalancer
- JMS Client
- RBAC with SASL/PLAIN RBAC with no SSL encryption, SASL/PLAIN authentication
- Audit Logs
- Confluent REST Proxy Security Plugin with SASL_SSL and 2WAY_SSL Principal Propagation
Easily play with Confluent Platform Ansible playbooks by using Ubuntu based Docker images generated daily from this cp-ansible-playground repository
There is also a Confluent Cloud version available here
- Confluent Replicator [also with SASL_SSL and 2WAY_SSL]
- Testing Separate principals (
connector.client.config.override.policy) for Source connector (SFTP source) - Testing Separate principals (
connector.client.config.override.policy) for Source connector (SFTP sink) - How to write logs to files when using docker-compose
- Publish logs to kafka with Elastic Filebeat
Monitor Confluent Platform with Datadog
- A Kafka Story 📖: A step by step guide to use Kafka ecosystem (Kafka Connect, KSQL, Java Consumers/Producers, etc..) with Docker
- Kafka Boom Boom 💥: An attempt to break kafka
- Kafka Security playbook 🔒: demonstrates various security configurations with Docker
- MDC and single views 🌍: Multi-Data-Center setup using Confluent Replicator
- Kafka Platform Prometheus 📊: Simple demo of how to monitor Kafka Platform using Prometheus and Grafana.