Introduce partitioner interface

[mlowicki]

Add interface to specify custom partitioners by extending ProducerContext trait with capability to return optional custom partitioner.

[scanterog]

Thanks for contributing. I've reviewed this and I personally think it would be better to use an associated type instead of generic type for the Partitioner in the ProducerContext.

I've spent some time and created this patch so you can observe the difference. By doing that we avoid introducing the Partitioner trait pretty much everywhere. The patch might require further testing. It was mostly to illustrate the idea. Ideally we also want a separate CollectingContext context for testing the custom partitioner on the low level producer tests.

I'm going on vacation next week and I'll let you sync with @duarten and @davidblewett on what you all think is better for this.

[mlowicki]

Thanks for contributing. I've reviewed this and I personally think it would be better to use an associated type instead of generic type for the Partitioner in the ProducerContext.

I've spent some time and created this patch so you can observe the difference. By doing that we avoid introducing the Partitioner trait pretty much everywhere. The patch might require further testing. It was mostly to illustrate the idea. Ideally we also want a separate CollectingContext context for testing the custom partitioner on the low level producer tests.

I'm going on vacation next week and I'll let you sync with @duarten and @davidblewett on what you all think is better for this.

Yeah, so using associated type was my first approach (I've working version somewhere). Later on decided to use generic type because of few reasons:

• Associated type defaults are unstable (Tracking issue for RFC 2532, "Associated type defaults" rust-lang/rust#29661) so for every implementation you would need to add something like type CustomPartitioner = NoCustomPartitioner;. With generic types we can use default value like pub struct ThreadedProducer<C, Part: Partitioner = NoCustomPartitioner> so things will remain backward compatible. If we would go with associated types then it'll break for everyone - it won't be hard to fix it but still.
• It makes more sense to support more than one partitioner per ProducerContext. Let's take a very simple example like https://github.com/fede1024/rust-rdkafka/blob/master/tests/test_low_producers.rs#L71 (CollectingContext). Using Generic types allows for better code re-use.

For sure using generic types requires more changes in the code base + signatures are becoming more complex. I'll start a thread on Slack to discuss on the direction here.

[davidblewett]

+1