Kafka connection using SCRAM authentication

[TheDavidKruse]

Requirement - what kind of business use case are you trying to solve?

In our current architecture, we secure our connection to our Kafka pipelines using the SCRAM authentication method. We would like to use Go as our Jaeger collector for ease of use in microservices and to replace our current Java implementation.

Problem - what in Jaeger blocks you from solving the requirement?

The current Go collector doesn't have an implementation to authenticate with Kafka using SCRAM. This forces us into continuing the use with our Java version with higher CPU and memory overhead in our microservice.

Proposal - what do you suggest to solve the problem or improve the existing situation?

By using the xdg-go/scram package we can implement adequate SCRAM authentication between the collector and Kafka.

I have a pull request able to be made if it's applicable.

Any open questions to address

Would there be a better SCRAM auth package that could be used?
Would SCRAM with Kafka be useful for anyone else?

[yurishkuro]

Maybe this should be solved in Shopify/sarama? Jaeger simply uses what sarama provides, we don't build additional mechanisms which are not part of the driver.

[TheDavidKruse]

That makes sense, let me head over there and see what kind of trouble I can stir up.

Thanks @yurishkuro

[TheDavidKruse]

As it turns out, the way we have implemented it comes directly from the way they have in their (Sarama's) example.

However, with the example from their repo they do not take into account using Jaeger with Kafka as they have a custom consumer and producer loop. I find that the overlap between what they have implemented in their own way would be easy to implement into Jaeger as it takes care of most of the producer and consumer code and allows for a new authentication method next to TLS and Kerboros.

If it's not worth implementing, is the incorrect implementation location, or is not viewed as something that would benefit Jaeger directly I will close the issue.

[zenobiuszeto]

@yurishkuro The PR which we are going to raise will enable the support the SCRAM based SASL fro Jaeger collector for Kafka which uses Shopify/sarama.

Today we have the support for TLS and kerberos.

Are you aligned on raising the PR ?

[zenobiuszeto]

@yurishkuro I did this case during end of last year

is there a plan to enable SCRAM SASL from Jaeger collector side to kafka ?

Juraci Paixão Kröhling @jpkrohling Nov 21 2019 01:06
@zenobiuszeto what does it take to add support for it?
@zenobiuszeto if you can provide a PR, chances are high that we can merge it

[yurishkuro]

I am still not clear why sarama has this SCRAM client abstraction rather than implementing it directly, I asked on their PR IBM/sarama#1295 (comment).

[yurishkuro]

@jaegertracing/jaeger-maintainers how do we feel about taking a dependency on https://github.com/xdg-go/scram? That repo hasn't changed in 2yrs, a very straightforward (and the only) PR opened 6mo ago had no maintainer feedback.

[TheDavidKruse]

Just bumping to see if there were any decisions yet. Thanks!

[yurishkuro]

I added this to the agenda for the next project status call. Feel free to join. https://docs.google.com/document/d/1ZuBAwTJvQN7xkWVvEFXj5WU9_JmS5TPiNbxCJSvPqX0/edit#

[TheDavidKruse]

Thank you!

[vprithvi]

We may also want to consider alternatives like confluent-kafka-go or confluent-java (via gRPC plugin framework), which are under active development and seem to support SCRAM.

[yurishkuro]

Decisions from status call:

• Accept xgd-go/scram for now
• Defer decision on Confluent until someone has time to test thoroughly

@TheDavidKruse do you want to propose a PR?

[TheDavidKruse]

@yurishkuro I can do that! Give me a day or two to propose a PR.

[Neustradamus]

The merged PR is:

• Add support for Kafka SASL/PLAIN authentication via SCRAM-SHA-256 or SCRAM-SHA-512 mechanism #2724