ApplicationFailedEvent should always quit application but jvm is kept alive by kafka consumers

[tnormani]

Hello,

I'm facing an issue on Spring Boot 2.5.4 under the following conditions:

• spring-boot-starter-parent:2.5.4
• spring-boot-starter-web:2.5.4
• spring-kafka:2.7.6
• embedded Tomcat won't start because Port 8080 is already in use

In this scenario an ApplicationFailedEvent is triggered and main thread is killed, but the application won't quit because kafka consumers threads are still alive. This results in a compromised application that appears like is up and running, but it is not.
I have also made a test using legacy spring-boot-starter-parent:2.2.10.RELEASE and related dependencies. In this scenario the behaviour is different and the Spring boot application quits as expected.

I've published a simple project to reproduce the problem at this link https://github.com/tnormani/application-failed-event-testcase .
Project requires port 8080 to be already bound by something else and a kafka broker for the consumer connection.
In the project you will find also a ApplicationFailedListener I created during tests, but it's not referenced in the spring.factories file to make test case behave as expected.

I've also raised a question on StackOverflow (https://stackoverflow.com/questions/70378200/how-to-make-a-spring-boot-application-quit-on-tomcat-failure) days ago but after the comparison with spring-boot-starter-parent:2.2.10.RELEASE and some further testing, I thought it would be better if you could have a look at it.

I've already opened an issue to Spring-Kafka on this (spring-projects/spring-kafka#2055) but they told me this a possible question for Spring Boot.

Thanks in advance.

[wilkinsona]

Thanks for the sample, I've reproduced the problem.

The change in behaviour is due to a change in Spring boot 2.3 where the web server's lifecycle is now managed via a SmartLifecycle implementation. When a SmartLifecycle implementation fails to start and Framework is cleaning up the application context as a result, it would appear that other previously-started SmartLifecycle implementations are not called to stop(). I need to investigate further but my initial impression is that this may be a Spring Framework bug.

[wilkinsona]

Looking at the javadoc of Lifecycle.stop(), I think this may, in fact, be a Spring Kafka bug:

on hot refresh during a context's lifetime or on aborted refresh attempts, a given bean's destroy method will be called without any consideration of stop signals upfront

What I have observed is that destroy() is called on KafkaListenerEndpointRegistry and its MLC isn't called as it doesn't implement DisposableBean. My reading of the javadoc for Lifecycle is that the registry needs to cope with a call to destroy() without stop() being called first and still clean itself up. I'll comment on spring-projects/spring-kafka#2055.