Queueing events when event volume exceeds capacity?

[ericstalbot]

This is a question, rather than a bug or feature request.

Is it possible to configure a max number of concurrent requests going out from the dispatcher to the subscriber?

Context: I have a use case where the volume of incoming events temporarily exceeds the capacity of the subscriber to process them. So I'd like events to be "queued up", and only delivered to the subscriber as it has capacity.

For example, I'd like to be able to say that there should be a maximum of 5 ongoing requests between the dispatcher and the subscriber at any given time. If the number of on going requests ever drops below 5, and there are more events in the queue, then a new request would be started to handle the next event.

Is something like this possible using the Kafka broker?

[pierDipi]

Hi @ericstalbot, I summarized the config you can change to get your desired behaviour in #2597, in particular, #2597 (comment).

In your case, you would need to reduce them because you need the exact opposite.

Note, those config impact every Kafka trigger, so they are not very fine grained, if that's a problem for you, we should refer to this feature request: #1412 (contributions are welcome)

The other option is to switch to use ordered delivery for a particular trigger that you want to control parallelism for https://knative.dev/docs/eventing/brokers/broker-types/kafka-broker/#configuring-the-order-of-delivered-events and the parallelism would be based on the number of partitions for a Knative Kafka Broker.

[pierDipi]

For example, I'd like to be able to say that there should be a maximum of 5 ongoing requests between the dispatcher and the subscriber at any given time. If the number of on going requests ever drops below 5, and there are more events in the queue, then a new request would be started to handle the next event.

For this particular case, if you really need an exact parallelism of 5, the only option is to use ordered delivery with a Knative Kafka Broker with exactly 5 partitions.

[ericstalbot]

Thanks! This answers my question.