Other delivery mechanisms than Kafka

[gjermundgaraba]

I am quite interested in this solution, but I don't want to use Kafka. Are there any plans or thoughts around other delivery mechanisms?

[altbro]

I am quite interested in this solution, but I don't want to use Kafka. Are there any plans or thoughts around other delivery mechanisms?

Hi!
we mainly focused to deliver along current architecture, but we tried to develop crawler with modular interfaces, so it shouldn't be very difficult to adopt another broker.
But it would be interesting why kafka is not ok? what are the downsides or it just some specific project requirement ?

[gjermundgaraba]

I understand that. Glad to hear it is designed to be modular.

I just don't particularly like Kafka. It is heavy in both the operational and learning sense. I simply don't want it in my stack. Even if I can run it myself, I don't want my team to spend time to learn it and run it.

I think it might have been wise to start with something more lightweight that smaller teams could more easily pick up.

In fact, I believe the best solution for something like this would be to be agnostic to the delivery method. Providing a simple interface to implement might have been more powerful, IMO.

[altbro]

Thank you for sharing your point @gjermundgaraba !
I probably should mention that kafka plays intermediate role in process, and we try to provide initial config for it to work out-of-box, so potential operators wouldn't need to mess with in-depth learning of what's under the hood. At the end only db matters to end-consumer.
But, would you be able to provide an example (or maybe even couple of them) of what may suit us instead of Kafka?

[sascha1337]

kinda agree with @gjermundgaraba - also you might consider this too:
bro-n-bro/spacebox#6