add support for on-demand topic metadata fetch

[andyxning]

This PR adds support for on-demand topic metadata fetching. For now, a KafkaClient will fetch metadata for all topics in the initialization. This will have some performance problem when run with a large Kafka cluster.

Adding a topics (type set) key-value parameter to KafkaProducer. This should be all the topics that send will be invoked on. For example, if send will be invoked on send('a', ...) and send('b', ...), then the value for topics will be {'a', 'b'}.

---

This change is 

[andyxning]

/cc @jeffwidman @dpkp @mumrah @mahendra @wizzat @vshlapakov

[dpkp]

I see the problem you're trying to solve and understand your approach here, but I'm not excited about the additional conceptual complexity here. I think the broker version probing issue could be avoided for now by setting an explicit api_version . And I wonder if we should just stop asking for topic metadata during _bootstrap at all. The java client, for example, "bootstraps" locally by creating fake broker objects derived from the bootstrap.servers list. As I understand the official client, it does not actually make a MetadataRequest until a Consumer or Producer action requires it for a specific topic (or all topics when consuming with a pattern match, etc). So I think it might be better to consider following this path. As a first step we could not request any topics when that is supported (MetadataRequest v1+). And then later we might think about how we could change the bootstrap process entirely to not make a MetadataRequest and just populate internal data structures based on the bootstrap_servers data. What do you think?

[andyxning]

And I wonder if we should just stop asking for topic metadata during _bootstrap at all. The java client, for example, "bootstraps" locally by creating fake broker objects derived from the bootstrap.servers list. As I understand the official client, it does not actually make a MetadataRequest until a Consumer or Producer action requires it for a specific topic (or all topics when consuming with a pattern match, etc). So I think it might be better to consider following this path.

This is a good question. AFAIK, sarama will not request any topic metadata before a real producer send operation. I like the lazy initialization.

I think the broker version probing issue could be avoided for now by setting an explicit api_version

This is not a doable way to resolve this problem. This just works around it. In case we operate many different Kafka version clusters, it is hard to let the users set api_version correctly themselves.

As a first step we could not request any topics when that is supported (MetadataRequest v1+). And then later we might think about how we could change the bootstrap process entirely to not make a MetadataRequest and just populate internal data structures based on the bootstrap_servers data. What do you think?

I agree with this approach. But it may take a long time, imo.

but I'm not excited about the additional conceptual complexity here.

Yes. i also think it is not so grace. But it is implemented optionally and have no backward incompatibility. I think it can resolve the problem quickly and even later we support to no do any metadata prefetching, it should be alright and no backward incompatibility will be made.

@dpkp WDYT?

[andyxning]

Ping @dpkp

[andyxning]

Ping @dpkp

[andyxning]

ping @dpkp

[dpkp]

Sorry - Ive been quite busy. I hope to take another look over the weekend.

…

On Thu, Jul 19, 2018, 9:19 PM Ning Xie ***@***.***> wrote: ping @dpkp <https://github.com/dpkp> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1541 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAzetFpcx53z4Qv_DmOS8aVMTTnEe4IUks5uIVpDgaJpZM4VOaJh> .

[andyxning]

@dpkp Big thanks for your answer and waiting for your comments. :)

[andyxning]

Ping @dpkp

[andyxning]

Ping @dpkp

[andyxning]

Sorry to periodically ping on this PR. @dpkp

[andyxning]

Ping @dpkp

[andyxning]

Ping @dpkp

[andyxning]

Ping @dpkp

[dpkp]

If we're going to land this I think we should name the config something more specific. Perhaps bootstrap_topics_filter .

[andyxning]

@dpkp Done. Please take a look at this.

[dpkp]

Reviewed 1 of 3 files at r1, 2 of 2 files at r2.
Reviewable status: complete! all files reviewed, all discussions resolved

[andyxning]

@dpkp Big thanks!