Combine blocking and non-blocking error handling

[ghost]

As before discussed here on Github.

Expected Behavior

A combination of blocking and non-blocking retry should be possible to handle three different types of exceptions:

• non-blocking (but not fatal) exceptions should be retried.

• non-blocking (but fatal, e.g. DeserializationException) exceptions should be routed to the DLT directly.

• blocking exceptions (e.g. a database is currently not available) should not be routed to a retry topic since all following messages would be routed there as well. The consumer should remain at the current offset.

Current Behavior
It is possible to use @RetryableTopic annotation to satisfy the first two bullets. It is also possible to use DefaultErrorHandler to implement the last bullet. But a combination of both approaches cannot be configured at the moment.

Context
@garyrussell mentioned a workaround by stopping the container using the stopImmediate container property. Nevertheless, there are a few blog articles that address the different handling of error types, e.g., this article. Therefore, having a proper way to configure this behavior would be a useful possibility.

[tomazfernandes]

Thanks for the suggestion @mariecuriiee, that's a good article.

I'm currently working on a couple of issues that should pave the way for this feature to work, by leveraging the existing DefaultErrorHandler we already set for the RetryableTopics. So hopefully we'll have this implemented soon.

[tomazfernandes]

Hi @mariecuriiee, I've added a PR with this feature, maybe you want to take a look:
#2124

LMWYT, thanks

[ghost]

Hey @tomazfernandes, thanks for implementing this so quickly! If I understand the implementation correctly, I only have to set the backoff using setBlockingRetryBackoff and add a classification behavior using setErrorHandlerCustomizer? And with setErrorHandlerCustomizer I could then define the behavior depending on the exception, e.g., for some exceptions the records go directly to the retry topics, for other exceptions, the records are handled using blocking retries or for other exceptions, the records are routed directly to the DLT?

[tomazfernandes]

Hi @mariecuriiee, thanks for looking into the solution. We’re currently reviewing it to improve the classification for blocking retries and make it allowlistable instead of the current denylist only behavior.

Either way, you’d have to configure the classification for each type of retry, i.e. define which exceptions are fatal for the blocking retries, and which are fatal for the non-blocking retries.

So if you want a non-default fatal exception to go straight to the DLT, you’d have to add it to both classifications. With this you can have any behavior you want - both retries, only one or another, or no retries.

We’re also in the process of reviewing the configuration API for the retryable topics feature as a whole, so when that comes out we’ll probably have a nicer API for configuring this.

As a side note, we’ve just introduced a way of setting global fatal exceptions for the non-blocking retries, maybe that’s something you’ll want to check out too.

[ghost]

@tomazfernandes that makes sense, thanks for clarification!
I thought about Gary's comment again and I agree that it would be more intuitive like he described. Even though this means that this feature won't be released in 2.8.3 😢

But as soon as it is released, I will use it in my project 😄 I also had a quick look at the global fatal exceptions - will test that too!

[tomazfernandes]

@mariecuriiee, we’ve implemented the solution, don’t know if you had the chance to take a look. It’s on 2.8.4-SNAPSHOT, and referenced in the docs: https://docs.spring.io/spring-kafka/docs/2.8.4-SNAPSHOT/reference/html/#retry-topic-combine-blocking

Thanks again for the suggestion, and feel free to provide any feedback.

@garyrussell, DYT there’s any more work to be done regarding this issue, or maybe we could close it?

Thanks

[garyrussell]

Thanks @tomazfernandes ; closing; resolved.