Support PubSub for Kafka

[c-thiel]

No description provided.

[Grongrilla]

I would like to give this one a shot

[Grongrilla]

Turns out there is something to discuss right from the get go.

I kind of expected that there would be one clear choice for a kafka rust lib, but at least at first glance, there is not.

There seem to be two more or less mature implementations available:

• kafka-rust
• rust-rdkafka

kafka-rust

Seems to be a pure rust implementation. A first look at the examples shows that it seems to be pretty easy to use. There a few things to mention, though

• Last release is 0.10.0, which is 11 months old. It has no tag in the GitHub repo (last tag is 0.9.0 from may 2022). Repo seems to be active, though
• They do not regard their lib as production ready: "Use it in production at your own risk."
• There seems to be no async client/producer implemented

rust-rdkafka

... is actually "just" a safe interface to librdkafka

• Last release is 0.36.2, which is 7 months old< librdkafka of course is actively maintained, the last release is 2 months old.
• async producer available
• They do not mention if the lib is production ready, however since it is based on librdkafka, one might assume that it should be
• rust-rdkafka can be enabled for Cloud Events SDK via feature flat. I am not sure if this is relevant, as the same is true for nats, however that feature flag is not used in TIP
• Reliance on a c lib has of course caveats.
  • librdkafka can be linked statically when the lib is build, using cmake (which is required on the build system); if SSL is required, libssl-dev is required.... and so on
  • librdkafka could be linked dynamically, introducing a dependency on the system that is running TIP

what next?

I am not sure what is the best choice here. If introducing a c lib is not an option, rust-kafke seems to be the only choice. If it is ok to schlepp around a c lib, rust-rdkafka is also async and seems to be "endorsed" by Cloud Events SDK.

Or maybe I am overlooking "that other kafka rust lib", that has less downsides than kafka-rust or rust-rdkafke 😄

[twuebi]

I went looking and found this, it's rather new but looks promising?

https://www.reddit.com/r/rust/comments/1ehpjgh/rust_native_kafka_protocol_and_client/ https://github.com/CallistoLabsNYC/samsa

[twuebi]

In terms of maturity & user-base it probably makes sense to stick to rdkafka for now, eventually we should switch over to a rust-native implementation to get rid of the C dependency.

[Grongrilla]

@twuebi

samsa indeed looks promising, but from your second comment I gather: rdkafka it is, for now.

Three questions:

• use the feature flag on cloud events sdk to pull in rdkafka, or add the dependency our self?
• Build librdkafka or depend on an existing version?
• which "extra features" (ssl, ...) are required?

I'd probably vote

• Bring in dependency ourself, which is consistent with how nats is used
• Build it our self
• I guess SSL is a must, no idea about the rest

[twuebi]

I'd say let's give rdkafka a try then, we should probably depend on cloudevents sdk's packaged rdkafka, from a cursory read, it seems that serialization of cloudevents to kafka is a bit more involved than what we do for nats, compare cloudevents-sdk-0.7.0/src/binding/nats/serializer.rs:19 with cloudevents-sdk-0.7.0/src/binding/rdkafka/kafka_producer_record.rs:24.

We've gone for depending on async-nats directly since cloudevents didn't package async-nats IIRC.

[twuebi]

Existing publishers can be found in crates/iceberg-catalog/src/service/event_publisher.rs:166..

[Grongrilla]

@c-thiel @twuebi

I just realized, that the latest release of cloudevents sdk depends on rdkafka ^0.29. Current release is 0.36.2.

0.29 is almost 2 years old. It depends on librdkafka 1.9, which is also almost 2 years old. Current version of librdkafka is 2.5.

The main branch of cloudevents sdk is already on ^0.36

Tbh, I am not sure what would be a good way to solve this 🙈

• If we do not bring in the feature, we loose the serialization for cloudevents to kafka and have to implement it ourselfs.
• ... or we could get cloudevents sdk from github, depending on the main branch.

[twuebi]

Hm, unfortunate, I'd say either ask CloudEvents-sdk for a release or vendor their serialization code for rdkafka

[Grongrilla]

If venodring is an option, I will do that. I can continue (well, start...) working and should also make things easier if or when cloud events sdk release a new version.

Regarding asking cloud events sdk for a release: maybe something you could or should do @twuebi? I'd maybe feel a bit uncomfortable since this is not my codebase 😅

[twuebi]

then let's start with vendoring