KSML is a wrapper around Kafka Streams that allows for development of low code stream processing applications. It was developed by Axual early 2021 and released as open source in May 2021.
Kafka Streams has captured the hearts and minds of many developers that want to develop streaming applications on top of Kafka. But as powerful as the framework is, Kafka Streams has had a hard time getting around the requirement of writing Java code and setting up build pipelines. There were some attempts to rebuild Kafka Streams, but up until now popular languages like Python did not receive equally powerful (and maintained) stream processing frameworks. KSML provides a new declarative approach to unlock Kafka Streams to a wider audience. Using only a few simple basic rules and Python snippets, you will be able to write streaming applications in very little time.
To quickly jump to the KSML specification, use this link: https://axual.github.io/ksml/
The project is divided into modules based functionality in order to be included separately depending on the use case.
The submodules are as follows:
| Module | Description |
|---|---|
ksml-data |
contains core data type logic, including BINARY and JSON types, used in Kafka Streams topologies and by Producers. |
ksml-data-avro |
extension to the data library for AVRO support. |
ksml-data-csv |
extension to the data library for CSV support. |
ksml-data-soap |
extension to the data library for SOAP support. |
ksml-data-xml |
extension to the data library for XML support. |
ksml |
the core component that parses KSML definitions and converts them to a Kafka Streams topology. |
ksml-kafka-clients |
the set of Kafka clients for KSML, injected into Kafka Streams, allowing for namespaced Kafka installations. |
ksml-query |
allows an active KSML application to be queries via REST for its internal state stores. |
ksml-runner |
standalone Java application for running KSML definitions. |
Building and running KSML requires an installation of GraalVM and the corresponding Python module. There are two ways to do this:
- Use the supplied multistage Docker build file
- Install GraalVM locally
See the paragraphs below for details.
You can build either the standard KSML runner, or the runner for the Axual platform using one of the following commands:
# Create the BuildX builder for KSML
docker buildx create --name ksml
# Build KSML Runner
docker buildx --builder ksml build --load --platform linux/amd64,linux/arm64 -t axual/ksml:local --target ksml -f Dockerfile .
# Remove the BuildX builder for KSML
docker buildx rm ksml
If you get the following error it means that your setup cannot build for multiple platforms yet.
ERROR: docker exporter does not currently support exporting manifest lists
You can perform a build for just your platform by removing the --platform linux/amd64,linux/arm64 arguments from the
commands above
# Create the BuildX builder for KSML
docker buildx create --name ksml
# Build KSML Runner
docker buildx --builder ksml build --load -t axual/ksml:local --target ksml -f Dockerfile .
# Remove the BuildX builder for KSML
docker buildx rm ksml
Download GraalVM for Java 21 or later from this page and install it for your platform as explained.
Once installed, select GraalVM as your default Java JVM. Then you can build KSML using the normal Maven commands:
mvn clean package
To run the KSML demo, we provide a Docker compose file which will start Kafka, create the demo topics, and start a container with a demo producer. You can then start the runner you generated in the previous step, passing in a KSML configuration of your choice. See Getting started or Runners for details.
To run the demo, Docker 19.x is required.
Axual is interested in building the community; we would welcome any thoughts or patches. You can reach us here.
See Contributing.