[QUERY] Error Handling in an Event Hub consumer - Best Practices

[remoba]

Library name and version

Azure.Messaging.EventHubs 5.7.0

Query/Question

I am wondering how I should go about handling failures in an Event Hub consumer. In this issue (18344) on the Java SDK repository James explains that Event Hubs don't have the concept of a dead-letter queue because messages are not destroyed upon consumption. Therefore, the consumer can then independently keep track of failed messages and retry them whenever it wishes.

What would a realistic best practice be? I imagine everyone who ever had to implement an Event Hub consumer has encountered this same question, and had to invent their own solution. The most basic solution I can think of is to leverage the already-existing checkpoint storage account and create 2 Azure queues in it.

1. In queue 1 I will store identifiers (Partition Id, Offset, Sequence Number) of messages that I have retried too many times
2. queue 2 would be a replay queue that I would listen to IN ADDITION to the Event Hub
3. An external process would then move messages between queue 1 and queue 2. It can be smart about it and only move messages that failed on non-permanent errors.
4. If the consumer fails on processing a message in queue 2 too many times, the messages will be moved back to queue 1

My questions are:

1. Is there a better recommended way to handle processing failures?
2. If not, does the Event Hub SDK expose a random access API to allow me to listen on queue 2 like that? I know the client allows to listen to a partition STARTING from a given Offset, but that's not really what I am looking for. I simply want to fetch a message by it's identifier to process it specifically (without any checkpointing)

Environment

No response

[jsquire]

Hi @remoba. Thank you for your feedback. There are a fair number of different messaging patterns for reliable processing of event streams; it's difficult to generalize what is "better" as different applications will benefit from different approaches. I'd recommend taking a look at the Azure Architecture Center and potentially the messaging patterns section. You may also want to take a look at Clemens Vasters' messaging patterns repository.

does the Event Hub SDK expose a random access API to allow me to listen on queue 2 like that

No; Event Hubs represents a persistent message stream. You can open a reader at a specific point in the stream but can only read forward in a contiguous manner. If you wanted to jump around to specific messages, you'd have to do so by opening and closing a reader each time - which will be a slow process.

[ghost]

Hi @remoba. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

[ghost]

Hi @remoba, since you haven’t asked that we “/unresolve” the issue, we’ll close this out. If you believe further discussion is needed, please add a comment “/unresolve” to reopen the issue.