Auto configuration for 2.9

[tomazfernandes]

I wasn't too sure as to where to bring this up - since 2.9 is a bootless version I thought of bringing this up here and we can move it to Boot if necessary.

The code purge we did for 2.9 has likely broken autoconfiguration for that version just as much as it did for 3.0. While for 3.0 we should be able to fix it, I don't see a clean solution for 2.9 since the correspondent boot version isn't aware of the new bootstrap classes.

If we're interested in fixing this, maybe we could create a workaround by adding code to the next boot release so it can bootstrap the feature itself, but it might be tricky to do it in a manner that is also compatible with 2.8.

Perhaps a simpler solution would be adding to the docs that for autoconfiguration to work with 2.9 users need to add the annotation themselves. Since it's an experimental feature that might not be too bad and users would still be able to benefit from the auto configured configuration bean. Doesn't look so clean either, but at least it doesn't require workaround code.

Of course, that's at your discretion - let me know if there's anything I can help with.

Thanks.

[artembilan]

From Spring Boot perspective it is indeed a breaking change.
We have claimed that 2.9.x is a drop-in version instead of 2.8.x for Spring Boot 2.7.x.

Therefore we have to look what should be brought back to make existing auto-configuration working without any changes.

The 3.0 story might be a bit different and it would be really better to realign auto-configuration with the current spring-kafka-3.0 API.

I think all of this related to the spring-projects/spring-boot#31620, however I'm not sure yet in details what is wrong and how to be...

[tomazfernandes]

Therefore we have to look what should be brought back to make existing auto-configuration working without any changes.

Sure, makes sense. Likely most of the code that was purged, including the RetryTopicBootstrapper, bean names, etc.

The 3.0 story might be a bit different and it would be really better to realign auto-configuration with the current spring-kafka-3.0 API.
I think all of this related to the spring-projects/spring-boot#31620, however I'm not sure yet in details what is wrong and how to be...

All auto-configuration does for now is add a RetryTopicConfiguration bean to the context. Once this is detected in the KLABPP, we'll try to fetch the RetryTopicConfigurer bean from the context to setup the feature for that @KafkaListener.

Up until 2.8, this triggered the RetryTopicBootstrapper logic, and the feature's infrastructure beans would be setup automatically.

With the 2.9 / 3.0 changes this is no longer the case and the RetryTopicConfigurationSupport has to be added manually by the user - unless we add such Support class to autoconfiguration in 3.0 as Andy suggested.

[artembilan]

Correct. We do that for 3.0 indeed. This is fully similar what is there for Spring MVC in the auto-configuration, for example.
For current Spring Boot 2.7.x, I'd prefer to not do any Spring Boot changes.
We have to do a change in spring-kafka-2.9.x to make it compatible with that 2.7.x.

[tomazfernandes]

One thing that comes to mind is, up until 2.8, all users had to do to use the feature was declare a RetryTopicConfiguration bean or @RetryableTopic annotation.

Now for 3.0 with autoconfiguration we can decide whether we want:

• for this to be a explicitly opt-in feature - users would have to either add the @EnableKakfaRetryTopic annotation or set the retrytopic.enabled=true property on boot to use it;
• to bootstrap the feature in autoconfiguration regardless of if users are using it or not, so that users can simply declare the configuration in the code and it will work, like in 2.8. In the end, it's only a couple of beans and a few objects...

Considering the majority of users probably doesn't use the feature, I guess the proper way would be the first approach and optimize for the most common use-case. But of course, I'm again biased and would be happy with the second 😄

Anyway, it's something I think we need to decide before fixing it for 3.0.

WDYT?

[artembilan]

Sorry, I'm not sure how is this relevant to the problem that we have removed some beans which are expected yet by Spring Boot.
I guess it is better to fix that part before going forward with possible auto-configuration overhaul.
Or do I miss anything?

Perhaps a requirement for the @EnableKafkaRetryTopic is not so bad: we do something similar already for Kafka Streams: https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#messaging.kafka.streams.
The feature is really auto-configured only if end-user provides an explicit @EnableKafkaStreams, or (if you wish) the KafkaStreamsDefaultConfiguration .
Therefore in our case we do require a RetryTopicConfigurationSupport and setup it from the auto-configuration.
Again: something similar what we do so far for Kafka Stream.
Does it make sense?

[tomazfernandes]

Sorry, I'm not sure how is this relevant to the problem that we have removed some beans which are expected yet by Spring Boot. I guess it is better to fix that part before going forward with possible auto-configuration overhaul. Or do I miss anything?

Well, I'm thinking we're bringing the legacy bootstrapping code for 2.9, but not for 3.0 - if that's correct I think they're really separate issues, right? Unless we're bringing it back for both to remove it at a later time in 3.0, might be ok too.

Perhaps a requirement for the @EnableKafkaRetryTopic is not so bad: we do something similar already for Kafka Streams: https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#messaging.kafka.streams. The feature is really auto-configured only if end-user provides an explicit @EnableKafkaStreams, or (if you wish) the KafkaStreamsDefaultConfiguration . Therefore in our case we do require a RetryTopicConfigurationSupport and setup it from the auto-configuration. Again: something similar what we do so far for Kafka Stream. Does it make sense?

Sure, seems reasonable.

I'm not that familiar with Kafka Streams to understand the impact of it being autoconfigured - I'd guess if users have the dependency on their classpath, they'd probably want to use it, so I'd probably default to autoconfiguring it rather than not. So I'm sure I'm missing something as for why it was decided in the other direction.

For the retryable topics, it's hard for me to measure the impact of bootstrapping it for users that will not use. Looking at a pure Spring Kafka point of view it doesn't seem like much, but from Spring Boot's perspective I think if every project keeps adding unnecessary beans that would be bad for the application as a whole.

Of course, from the usability side of things I'd prefer to have as less friction as possible for users. But adding the annotation doesn't really sound like a big deal, and we can always revise this later if necessary.

[artembilan]

Yes. they are two different stories.

I prefer to have 2.9.x to be realigned with 2.8.x support in Spring Boot without any changes in Spring Boot by itself.
As I said in the beginning of our discussion: just drop-in like we suggest for anything from Spring for Apache Kafka perspective: https://docs.spring.io/spring-kafka/docs/2.9.0-RC1/reference/html/#update-deps.
I know, this one is hidden from Spring Boot yet, just because you have to explicitly override to this version in the current Spring Boot 2.7.x.
But this will raise one day. Fortunately we have some time before 2.9.0 GA to bring those legacy beans back to make Spring Boot 2.7.x working again.
So, this is one story which we can discuss in its own issue.
(Perhaps this is my fault to agree to remove the legacy API during review of that PR...)

Another problem is Spring Boot 3.0 support.
This one probably is broken at the moment because it depends on spring-kafka-3.0.0-SNAPSHOT. That's why Andy points from that Spring Boot issue.
In my feeling this is less urgent than 2.7.x compatibility, but it still has to be done in time for next Spring Boot milestone in a couple weeks: https://calendar.spring.io/.
And here I think we have a freedom to rework that auto-configuration for retry topics as we prefer with our current API.

Does it all make sense?

P.S. Yes, I also cannot get it why an optional kafka-streams jar is not enough as a trigger for respective auto-configuration. But looks like a decision was made to be based on an explicit end-user willing via that @EnableKafkaStreams.

[tomazfernandes]

I prefer to have 2.9.x to be realigned with 2.8.x support in Spring Boot without any changes in Spring Boot by itself.

Sure, makes sense.

(Perhaps this is my fault to agree to remove the legacy API during review of that PR...)

Well, I guess that's what these pre-release versions are for, right? I didn't realize it'd break autoconfiguration either. Not sure many people are using it since it's not documented in SK's docs. Maybe we should add it?

In my feeling this is less urgent than 2.7.x compatibility, but it still has to be done in time for next Spring Boot milestone in a couple weeks: https://calendar.spring.io/.

And here I think we have a freedom to rework that auto-configuration for retry topics as we prefer with our current API.

Sure, sounds good. And being for 3.0 we'll have time to review any decision in the next milestones.

P.S. Yes, I also cannot get it why an optional kafka-streams jar is not enough as a trigger for respective auto-configuration. But looks like a decision was made to be based on an explicit end-user willing via that @EnableKafkaStreams.

Is it a decision by the Boot team? Maybe we can ask them if so.

[garyrussell]

I guess we have to reinstate the bootstrapper, at least in 2.9, but maybe it's ok in 3.0 as well (but maybe we should make the Bootstrapper package protected).

[garyrussell]

I think I have another solution. Stand by...

[tomazfernandes]

Thanks @garyrussell, looking forward to seeing it.

Just my 2c after some consideration - for 3.0, we currently have the retry-topic-enabled property in auto-configuration - we could lean on that to import the ConfigurationSupport on behalf of the user.

If the property is not provided (no auto-configuration for the feature), users would have to explicitly use the @EnableKaffaRetryTopic annotation or subclass and import the Support class.

Of course, a solution where users might simply declare a RetryTopicConfiguration bean or @RetryableTopic and get the feature working would be preferable, but I prefer that than having both bootstrapping logics in 3.0.

[garyrussell]

#2343

I was a bit surprised to find that Boot auto configuration disables auto creation of topics.

[tomazfernandes]

#2343

This looks good, and I think it's a great way of leveraging the new configuration system while still providing automatic bootstrapping of the feature. Mind that when bootstrapping this way users may not be able to inject beans such as the DestinationTopicResolver, as we've seen to be requested in the past.

TBH I'm not sure how I feel about having so many possibilities for starting and configuring the feature - autoconfiguration, annotation, importing a subclass, and now automatic bootstrap. Perhaps we could try to streamline this a bit? But it might be ok too, and we can always revise it before 3.0 is out.

I was a bit surprised to find that Boot auto configuration disables auto creation of topics.

We can revert this in 3.0 - I thought creating topics automatically would be too invasive for default auto-configuration, and since we're being minimalistic there I preferred being conservative. In production environments apps will rarely have permission to access the Admin ports in Kafka, so if I'm not mistaken we might even have an error thrown if we left it as default without an option to disable.

If you have any other thoughts on what should be the sensible defaults for autoconfiguration we can change that for 3.0 as well!

[garyrussell]

I guess my surprise was more that it cannot be changed.

I don't really object to it being false by default, but it should at least be configurable via properties/yaml.

To work around it in my test, I added @RetryableTopic(autoCreateTopics = "true", fixedDelayTopicStrategy = FixedDelayStrategy.SINGLE_TOPIC) to my listener and ran it once; then removed it.

users may not be able to inject beans ...

I consider this as a last resort and should never be needed; our documentation (now) clearly states that they need to extend RTCS in one of their @Configuration classes.

[tomazfernandes]

I guess my surprise was more that it cannot be changed.

I don't really object to it being false by default, but it should at least be configurable via properties/yaml.

Sure, we can add that option after we've merged this.

To work around it in my test, I added @RetryableTopic(autoCreateTopics = "true", fixedDelayTopicStrategy = FixedDelayStrategy.SINGLE_TOPIC) to my listener and ran it once; then removed it.

In the meantime, with the current defaults, for fixed delays only one retry topic + dlt are needed: "my-topic-retry-topic" and "my-topic-dlt", so shouldn't be too hard to create the couple of "NewTopic" beans if necessary.

For exponential defaults, the names are suffixed by index with one topic per attempt after the first, so for 3 attempts should be "my-topic-retry-topic-0", "my-topic-retry-topic-1" and "my-topic-dlt".