Message delivery latency with batching enabled

[hansenc]

I'm trying to calculate message delivery latency, which is the time between a message getting published until the message being delivered to a consumer. I have batching enabled.

Normally, I would calculate this by dividing backlog size by throughput. However, batching throws off this calculation because I can only find out the number of batches on the backlog rather than individual messages. I see a number of timestamps on the topic stats, but I'm unsure which, if any, can be used for this purpose.

[lhotari]

I don't have a solution, but some questions:

• with calculating the message delivery latency, is it about predicting the latency for the oldest backlogged message?
  • Is this about estimating the current lag?
    • Who needs this information and what type of decisions and actions are made based on this information?

[hansenc]

Good questions. I am trying to provide an estimate to business stakeholders regarding how long it will take to process through all messages currently on the topic so they can understand when business value will be delivered. In my case, no real time decisioning is being made, though I'm sure there are other people in that situation. We've had some performance issues until very recently which have meant at times we have not been delivering business value quickly enough, leading us to question what alternative solutions might be possible.