Feature Request: Azure Event Hubs Output - Support setting partition key

[anthonysomerset]

Feature Request

Opening a feature request kicks off a discussion.

Proposal:

Current EventHub output works great - for some users there is a need to control how events get sent to specific partitions for event ordering or other similar processing requirements

The Azure Eventhubs Go module which the output plugin uses suggests that this could/should be as simple as setting the PartitionKey value for an event and then eventhub will take care of hashing based on that key and consistently send events with that particular value to the same partition. - https://github.com/Azure/azure-event-hubs-go#send-and-receive

I propose that we add some configuration that allows the user to configure which value to set for this based on a specific Tag or data field with the default to omit this entirely which would retain current behaviour by default.

e.g.

partition_key_field = "SubscriberId"

Current behavior:

No partition key value is set on events meaning all events are randomly load balanced over eventhub partitions

Desired behavior:

PartitionKey value gets set on events so that they are consistently load balanced over eventhub partitions in a deterministic manner

Use case:

This is useful because by default downstream processing can then take advantage that specific events will group into specific partitions and can perform relevant optimizations accordingly. an example in this scenario is Azure Stream Analytics Jobs

Also important to note is that some events may be very order dependent - being able to control events going to specific partitions allows for order of events to be guaranteed in subsequent processing

[powersj]

Hi,

Per the link you provided and Azure/azure-event-hubs-go#134 it does look like setting the event's PartitionKey will send that event to a specific partition.

My concern however is how the partition key is determined. Your proposal is to provide a field/tag value to read from and set the partition key from there. This could result in a huge number of partitions getting generated. My understanding is while you can create more partitions to increase throughput, this might result in dozens, hundreds, etc. of partitions getting created if you are not careful. This does not seem ideal either.

Based on your suggestion of "SubscriberId" how many partitions would you be generating with your data?

I assume having a setting to allow a specific event_hubs output to all go to a specific partition is not enough?

Thanks

[anthonysomerset]

You won’t create more partitions than you configure on the eventhub side. That will always be a hard limit no matter the cardinality of the partition key value The partition key value gets hashed and that hash determines which of the available partitions the event will get sent to as the potential hash values get split evenly over the partitions It’s loosely equivalent to how load balancers will balance connections and perform stickiness

[powersj]

Ah, thank you for the explanation that does indeed help clarify the situation.

next steps: update plugin to add a partition_key_field option that optionally takes a field to read. If the field exists, then the partition key will be set with this value for each event before adding to the batch.

[powersj]

I put up #10795 as a quick test of what this might look like. If you want to grab one of the artifacts and test it, it would be very helpful.

[telegraf-tiger]

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Page. Thank you!