Using SSL with librdkafka

SSL support in librdkafka

The SSL support in librdkafka is completely configuration based, no new APIs are introduced, this means that any existing applications dynamically linked with librdkafka will get automatic SSL support by upgrading only the library.

This page serves as a simple HOWTO guide.

Also see Deploying SSL for Kafka

Prerequisites

• Apache Kafka brokers: version 0.9.0.0 or later.

• Make sure the openssl and libssl-dev packages are installed.

• Reconfigure and rebuild librdkafka (./configure --reconfigure && make). Verify that WITH_SSL is set to 1 in config.h

• Create a convenient ssl directory where you execute the commands to create certificates and keys.

• All key and keystore passwords are abcdefgh. See gen-ssl-certs.sh for how to change this.

There is a script in librdkafka's tests/ directory called gen-ssl-certs.sh that automates the certificate and key generation steps outlined in the above link. It will be used throughout this HOWTO so make sure the script is in your $PATH (or equivalent).

Create a CA certificate

If you dont have a proper CA certificate you can generate your own for testing.

gen-ssl-certs.sh ca ca-cert <the_ca_CN>

Create broker keystore

For each broker (let $BROKER be broker hostname), do:

gen-ssl-certs.sh -k server ca-cert broker_${BROKER}_ ${BROKER}

Create client keys

This is only needed if you want to authenticate clients on the broker.

Create standard client keys (for librdkafka, et.al.)

The generated keys are standard OpenSSL PEM keys usable by librdkafka and any OpenSSL based client (and probably others as well).

For each client (let $CLIENT be client name), do:

gen-ssl-certs.sh client ca-cert client_${CLIENT}_ ${CLIENT}

Create client keystore for Java clients

This is only needed if you want to use the official Java clients that uses a Java keystore instead of standard PEM keys.

For each client (let $CLIENT be client name), do:

gen-ssl-certs.sh -k client ca-cert client_${CLIENT}_ ${CLIENT}

Configure broker

For each broker copy its keystore files (broker_${BROKER}_*.jks) to the broker node and add the following to the broker's server.properties configuration file (replace filenames as needed):

# SSL
ssl.protocol = TLS
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.keystore.type = JKS
ssl.keystore.location = broker_???????_server.keystore.jks
ssl.keystore.password = abcdefgh
ssl.key.password = abcdefgh
ssl.truststore.type = JKS
ssl.truststore.location = broker_????????_server.truststore.jks
ssl.truststore.password = abcdefgh
# To require authentication of clients use "require", else "none" or "request"
ssl.client.auth = required

Restart the brokers and monitor the log output to see that the configuration was accepted.

Configure librdkafka client

For each client copy its key files (client_${CLIENT}_*) and the public CA-cert to the client node and configure your librdkafka application with the following properties:

metadata.broker.list=at_least_one_of_the_brokers
security.protocol=ssl

# CA certificate file for verifying the broker's certificate.
ssl.ca.location=ca-cert

# Client's certificate
ssl.certificate.location=client_?????_client.pem

# Client's key
ssl.key.location=client_?????_client.key

# Key password, if any.
ssl.key.password=abcdefgh