No support for security

[orefalo]

Can't connect to a SASL_SSL SASL_PLAINTEXT or even PLAINTEXT authenticated cluster

[arbrick]

I would like to throw my hat in for ssl support. It would be great if it could be integrated.

[fgeller]

@orefalo @arbrick hey guys - thanks for the feedback and sorry for the delay. Do you guys have some recommendations / ideas for the interface or existing examples? Also a sample Kafka config would help that I could test against :) Ideally something I could use for a wurstmeister/kafka image :)

[orefalo]

sorry not close to this topic anymore. but there are plenty of examples online around kerberos, plaintext, sasl and ssl. Sincerely.

[arbrick]

@fgeller I am specifically interested in the ssl configuration. The wurstmeister/kafka image has a mention of the ssl configuration here, but it doesn't go into the details of setting it up. I am mainly familear with kafka from a consumer standpoint and don't have a whole lot of experience setting up the brokers. The Kafka documentation tells you which configs you need to specify.

[orefalo]

Now I am closer than ever... https://github.com/orefalo/docker-kafka-ssl

[tednaleid]

I've used this tls branch of a kt fork and it has worked out well. https://github.com/njhartwell/kt/commits/tls this code expects a single file with the CERTIFICATE and PRIVATE KEY but the code could be modified to take 2 files in. It adds a -tls flag and a --clientCert <public & private key> flags but otherwise works just like the existing kt app.

The clientCert flag is a little misleading as it also has the unencrypted private key in it too.

We've talked on my team about submitting a PR for this but have struggled with how to implement tests that are valid.

[hongkongkiwi]

+1 for this feature, it's hard to use in production without this.

[tsjnsn]

If you're using JKS you can try my fork: https://github.com/tsjnsn/kt

# JKS
$ kt consume -verbose -topic test -offsets oldest -keystore keystore.jks -keypass somepassword -keyalias clientCertAlias -caalias authorityAlias -brokers broker1:9092

# PEM
$ kt consume -verbose -topic test -offsets oldest -key keystore.pem -cert ca.crt -brokers broker1:9092

Still needs some work before a PR though

[iguyking]

https://github.com/confluentinc/cp-demo gives a docker-compose that comes up with security enabled. This can be used quickly as a basis for testing security configurations against kt

[fgeller]

linking #86

[fgeller]

hey everyone, quick update:

@rwaweber implemented functionality to use TLS to encrypt the connection to brokers, cf #86 and thanks again @rwaweber :)

had some time to read up on the different auth mechanisms that kafka and sarama support and would like to proceed like this:

1. generally offer support for what sarama supports. afaict, that's TLS certs and optional plaintext passwords. i'd hope that'd cover 80% of use cases and we could look at more special cases in more detail as need be.
2. simplify the way we provide auth details a bit, and make it easier to add more auth mechanisms later on. more specifically:

• add an -auth flag to each command that can also be provided via an KT_AUTH env var. it'd expect a json encoding of the auth details. i think that'd make it generally easier to keep the auth details in a file and set an environment variable once per session or so rather than adding the flags to each kt invocation. e.g. in the case of TLS the file could look something like this:

{ "type": "TLS", "key": "/path/to/key", "ca": "/path/to/ca", "certificate": "/path/to/cert" }

keen to get feedback as always

[orefalo]

Hi @fgeller

Good job!

Really like kt, it's neat.

Unfortunately, In my company, we don't use passwords.

We use 2 way SSL - IT Security policy!
there is a cert on each side, signed with the same parent key (kafka limitation as of 6 months ago).

Sincerely,

[fgeller]

Hi @orefalo - glad it's a tool that helps :)

Could you provide a code snippet, preferrably in Go [using sarama] that shows how you authenticate a client?

[tsjnsn]

Here's one: https://github.com/tsjnsn/kt/blob/master/consume.go#L346

[fgeller]

cheers @tsjnsn ! That looks a lot like what we're doing at the moment thanks to @rwaweber: setupCerts - besides the key store handling above the line that you link, or am I missing something?

[orefalo]

I am afraid I don’t code in Go. I use your tool from the command line ;-)

My stuff is here…
https://github.com/orefalo/docker-kafka-ssl

It generates the keys, stands up a server and has Java and node.js clients.

I guess GO will use a similar configuration to node.js (no Java specific JKS stuff).

Oli

[fgeller]

@orefalo fair enough :) should be able read the node.js bits - will give that a go tomorrow, cheers!

[cmaenner]

For all that are interested in using the tool to securely consume from topic, here is a quick oneliner that will consume from topic "foo" and will output pretty JSON from the value field using the jq application (not included with kt):

kt consume -brokers "broker01:9092","broker02:9092" -tlsca ca.pem -tlscert cert.pem -tlscertkey key.pem -topic "foo"  | jq '.value | fromjson'

[cmaenner]

I would also recommend this issue be closed now we have support for certificates

[cmaenner]

Also another resource on how to build certificates, if you want to test:

Reference

How to generate self-signed certificates

cfssl print-defaults config > ca-config.json
cfssl print-defaults csr > ca-csr.json
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
cfssl print-defaults csr > server.json
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server.json | cfssljson -bare server

[ravanscafi]

It's working like a charm :)

I just wish we could have env vars for those too 👍

[fgeller]

hey all -- it's been a while, sorry again for the delay - i have been a full-time dad for the past 2.5 years and only recently resumed work on my github projects. i've finally gotten around to setting up the system tests such that we have some automated testing for the authentication features and that allowed me to apply the changes i mentioned in the above comment more confidently. let me know if you run into any trouble by creating an issue with details please. also, if you are looking for a different authentication mechanism, please create a github issue with details - or add a pull request 😉