Sparse connections

[winbatch]

Let's say I have 5 broker hosts in the cluster. I'm noticing if I supply a single broker to the API, there will be 6 connections created. 1 to each broker and 1 extra to the broker I supplied the API. I imagine the logic is simple and it is connecting to all the brokers in the metadata list but I think it should ignore the 1 it's already talking to. Can it be enhanced to skip him? We are trying to reduce the number of connections to the brokers.

[edenhill]

A shortcoming of the Kafka protocol is that you can't identify the broker you are connected to, you can only do this by reference.
The reference is the metadata received from the bootstrap broker(s), which says:
broker id X is at host.example.com:9092

That's why librdkafka can't know that the bootstrap broker is indeed the same as one reported from metadata.

Anyway, others have raised concerns about the number of open connections, and I believe the scala/java clients only keeps connections open to the brokers it needs (e.g., the leaders for topic+partitions being produced/consumed).
I'm planning to add the same sparse behaviour to librdkafka, probably with a new configuration property that is disabled by default (!!).

[winbatch]

Hmm. Can't you just call getpeername() on the socket to know who you're talking to?

[edenhill]

I could, but DNS does not have to be asymetric, nor 1:1, so it couldn't really be trusted.

Disconnecting from brokers we dont need will solve your problem though, e.g., sparse connections, so that's what I'll go for.

[winbatch]

Fair enough. I prefer your solution anyway as it further reduced connections

[winbatch]

Hi- do you happen to know when you might implement this? I've made some changes to my code to feed you only one broker at a time (as needed) to reduce connections. If you add the sparse option soon (like within the week), I'll include that in my release. If you think it will be longer I'll just release with my change only (still a 40% improvement in our env)
Thanks!

[edenhill]

Sparse connections will go into the dev branch with a bunch of other changes, so it is most likely a month or so from being merged into master.

[edenhill]

Duplcate of #825