kafka - disable keeping metadata for all topics

[famarting]

Description

Small change to the config of the kafka client used in all kafka components

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #3247

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• Code compiles correctly
• Created/updated tests
• Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

[famarting]

seems like this config option cannot be used without providing a known list of kafka topics...

[artursouza]

seems like this config option cannot be used without providing a known list of kafka topics...

Can that be passed as a parameter? Maybe there are users that have that scenario.

[olitomlinson]

@artursouza @yaron2

I think its worth reconsidering if there is a possible implementation here.

For context : we have a use case where thousands of partitions on the Kafka cluster are related to Kafka Topics that are never going to be used by Dapr sidecars. So having Dapr load all this metadata on init and refresh it is highly wasteful.

[artursouza]

@artursouza @yaron2

I think its worth reconsidering if there is a possible implementation here.

For context : we have a use case where thousands of partitions on the Kafka cluster are related to Kafka Topics that are never going to be used by Dapr sidecars. So having Dapr load all this metadata on init and refresh it is highly wasteful.

I can consider one. Can you track your request in an issue? We can tag it and get some traction from the community.