Shutdown issue in SpringBoot application

[mnbattaglia]

I'm using a SpringBoot MVC application and I'm starting the LocalS3 instance in a configuration bean @PostConstruct and shutting it down in the @PreDestroy. When stopping the application I'm getting the following error:

Exception in thread "Thread-3" java.util.concurrent.RejectedExecutionException: event executor terminated
	at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:934)
	at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:351)
	at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:344)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:836)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817)
	at io.netty.channel.AbstractChannelHandlerContext.safeExecute(AbstractChannelHandlerContext.java:1165)
	at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:729)
	at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:560)
	at io.netty.channel.DefaultChannelPipeline.close(DefaultChannelPipeline.java:957)
	at io.netty.channel.AbstractChannel.close(AbstractChannel.java:244)
	at com.robothy.s3.rest.LocalS3.shutdown(LocalS3.java:139)
	at java.base/java.lang.Thread.run(Thread.java:833)

It seems that local s3 gets stopped when spring boot app stops, so I think there is no need to manually call the shutdown in this case.
This is not happening in a WebFlux app

[mnbattaglia]

Given the following line in the start() method:
Runtime.getRuntime().addShutdownHook(new Thread(this::shutdown));
Is there any reason to call the shutdown() manually? It seems that it fails if the shutdown was already called by the hook

[mnbattaglia]

I think the problem is, if I call shutdown() programmatically, when JVM stops it will run the hook and try to do a shutdown() again and that's where it fails. The thing is, I need to do the shutdown before the JVM stops while running the Spring Boot app. Otherwise, the java running process gets stuck. I don't know how we can get this working. Probably by making the shutdown hook optional during locals3 start

[Robothy]

Given the following line in the start() method: Runtime.getRuntime().addShutdownHook(new Thread(this::shutdown)); Is there any reason to call the shutdown() manually? It seems that it fails if the shutdown was already called by the hook

We use this hook to handle the stop signal from Docker. If you create several LocalS3 instance during test, then manually shutdown is required. We can check if the Netty channels and event executors are open in the shutdown() method to ensure they won't be closed multiple times. I'll add a PR to do that.

[Robothy]

This issue has been fixed since version 1.12.1

[mnbattaglia]

Thanks A LOT @Robothy for the quick fix

[mnbattaglia]

Hi @Robothy , although this seems to be working, we are still getting logs like these:

18:07:24  {"timestamp":"2023-09-26T21:07:24.864Z","level":"WARN","thread":"nioEventLoopGroup-3-1","logger":"io.netty.channel.AbstractChannelHandlerContext","message":"Failed to submit an exceptionCaught() event.","context":"default","exception":"java.util.concurrent.RejectedExecutionException: event executor terminated\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:934)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:351)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:344)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:836)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:328)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)\n\tat io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:308)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)\n\tat io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813)\n\tat io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)\n\tat io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasksFrom(SingleThreadEventExecutor.java:426)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:375)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:763)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:596)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"}
18:07:24  {"timestamp":"2023-09-26T21:07:24.865Z","level":"WARN","thread":"nioEventLoopGroup-3-1","logger":"io.netty.channel.AbstractChannelHandlerContext","message":"The exceptionCaught() event that was failed to submit was:","context":"default","exception":"java.util.concurrent.RejectedExecutionException: event executor terminated\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:934)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:351)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:344)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:836)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:283)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:411)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:376)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:305)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)\n\tat io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813)\n\tat io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)\n\tat io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasksFrom(SingleThreadEventExecutor.java:426)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:375)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:763)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:596)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"}

We are using a Spring Boot configuration that stops the local s3 on a @PreDestroy event. Previously, when doing this, we had the issue that both shutdowns (our own event + the JVM hook) made the application not to shutdown correctly and the process was still alive after that. Now it seems that it works fine but we are seeing those logs. We are ignoring them, but might be confusing (even if they are harmless)

[Robothy]

Reopen this issue for investigation.

[Robothy]

#76