Jersey threads can attempt to write after Netty's event loops are shut down

[spericas]

Under heavy load and when Jersey threads are blocked on I/O, it appears these threads can attempt to write to a connection after the Netty's event loops are shut down, resulting in a RejectedExecutionException thrown by Netty. As a result, an application may report multiple exceptions while shutting down.

[spericas]

Details of exception that can be reported multiple times:

helidon-mp: java.util.concurrent.RejectedExecutionException: event executor terminated
helidon-mp: 	at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926)
helidon-mp: 	at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353)
helidon-mp: 	at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346)
helidon-mp: 	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828)
helidon-mp: 	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818)
helidon-mp: 	at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:841)
helidon-mp: 	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:498)
helidon-mp: 	at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:183)
helidon-mp: 	at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95)
helidon-mp: 	at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30)
helidon-mp: 	at io.helidon.webserver.BareResponseImpl.writeLastContent(BareResponseImpl.java:281)
helidon-mp: 	at io.helidon.webserver.BareResponseImpl.completeInternal(BareResponseImpl.java:221)
helidon-mp: 	at io.helidon.webserver.BareResponseImpl.onComplete(BareResponseImpl.java:396)
helidon-mp: 	at io.helidon.media.common.MessageBodyContext$EventingSubscriber.onComplete(MessageBodyContext.java:354)
helidon-mp: 	at io.helidon.common.reactive.MultiMapperPublisher$MapperSubscriber.onComplete(MultiMapperPublisher.java:98)
helidon-mp: 	at io.helidon.common.reactive.EmittingPublisher.signalOnComplete(EmittingPublisher.java:183)
helidon-mp: 	at java.base/java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:815)
helidon-mp: 	at java.base/java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:799)
helidon-mp: 	at java.base/java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2185)
helidon-mp: 	at io.helidon.common.reactive.EmittingPublisher.complete(EmittingPublisher.java:146)
helidon-mp: 	at io.helidon.common.reactive.MultiFromOutputStream.complete(MultiFromOutputStream.java:201)
helidon-mp: 	at io.helidon.common.reactive.MultiFromOutputStream.close(MultiFromOutputStream.java:129)
helidon-mp: 	at io.helidon.webserver.jersey.ResponseWriter$1.close(ResponseWriter.java:79)
helidon-mp: 	at io.helidon.webserver.jersey.ResponseWriter.commit(ResponseWriter.java:161)
helidon-mp: 	at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:390)
helidon-mp: 	at org.glassfish.jersey.server.ServerRuntime$Responder.release(ServerRuntime.java:736)
helidon-mp: 	at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:442)
helidon-mp: 	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:261)
helidon-mp: 	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
helidon-mp: 	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
helidon-mp: 	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
helidon-mp: 	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
helidon-mp: 	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
helidon-mp: 	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
helidon-mp: 	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
helidon-mp: 	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
helidon-mp: 	at io.helidon.webserver.jersey.JerseySupport$JerseyHandler.lambda$doAccept$3(JerseySupport.java:283)
helidon-mp: 	at io.helidon.common.context.Contexts.runInContext(Contexts.java:98)
helidon-mp: 	at io.helidon.common.context.ContextAwareExecutorImpl.lambda$wrap$1(ContextAwareExecutorImpl.java:129)
helidon-mp: 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
helidon-mp: 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
helidon-mp: 	at java.base/java.lang.Thread.run(Thread.java:832)```

[spericas]

The scenario here is a Docker container that is stopped via a SIGTERM followed by a SIGKILL. Perhaps the SIGTERM is causing the event loop to shut down while the Jersey threads are still running --which may explain why they attempt and fail to write data to a connection. More investigation is needed.

[spericas]

PR #2505