io.netty.handler.codec.http2.Http2Exception$StreamException: Stream closed before write could take place

[pgus2024]

Seeing this issue while sending notification. The errors occurs intermittently. The service is deployed on AWS US East region.
We are using version : 0.15.4

Caused by: java.io.IOException: Stream closed before a reply was received
	at com.eatthepath.pushy.apns.ApnsClientHandler.<clinit>(ApnsClientHandler.java:85)
	at com.eatthepath.pushy.apns.ApnsClientHandler$ApnsClientHandlerBuilder.build(ApnsClientHandler.java:139)
	at com.eatthepath.pushy.apns.ApnsClientHandler$ApnsClientHandlerBuilder.build(ApnsClientHandler.java:92)
	at io.netty.handler.codec.http2.AbstractHttp2ConnectionHandlerBuilder.buildFromCodec(AbstractHttp2ConnectionHandlerBuilder.java:614)
	at io.netty.handler.codec.http2.AbstractHttp2ConnectionHandlerBuilder.buildFromConnection(AbstractHttp2ConnectionHandlerBuilder.java:589)
	at io.netty.handler.codec.http2.AbstractHttp2ConnectionHandlerBuilder.build(AbstractHttp2ConnectionHandlerBuilder.java:553)
	at com.eatthepath.pushy.apns.ApnsClientHandler$ApnsClientHandlerBuilder.build(ApnsClientHandler.java:146)
	at com.eatthepath.pushy.apns.ApnsChannelFactory$1.initChannel(ApnsChannelFactory.java:129)
	at com.eatthepath.pushy.apns.ApnsChannelFactory$1.initChannel(ApnsChannelFactory.java:97)
	at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129)
	at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112)
	at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:1114)
	at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)
	at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)
	at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463)
	at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115)
	at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:480)
	at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:89)
	at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:83)
	at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86)
	at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:339)
	at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:164)
	at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:125)
	at com.eatthepath.pushy.apns.ApnsChannelFactory.lambda$create$3(ApnsChannelFactory.java:175)
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
	at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

[pgus2024]

We have multiple apps and APNSClient is created once for each app and used till the pod is killed.
The way we instantiate the APNSClient is:

EventLoopGroup eventLoopGroup = new NioEventLoopGroup(4); // Initialized in constructor
---------------
ApnsClient apnsClient = new ApnsClientBuilder()
        .setConnectionTimeout(Duration.of(10000, ChronoUnit.MILLIS))
        .setIdlePingInterval(Duration.of(10000, ChronoUnit.MILLIS))
        .setEventLoopGroup(eventLoopGroup)
        .setConcurrentConnections(4)
        .setMetricsListener(metricsListener)
        .setClientCredentials(cert, password)
        .setApnsServer(server).build();

[pgus2024]

@jchambers Could you please help me

[pgus2024]

It usually happens in spikes... I also see the following warning before this error:

2024-06-05T13:31:29.364Z [thread=nioEventLoopGroup-11-4] loglevel=WARN logger=com.eatthepath.pushy.apns.ApnsChannelPool Failed to create channel java.io.IOException: Stream closed before a reply was received at com.eatthepath.pushy.apns.ApnsClientHandler.<clinit>(ApnsClientHandler.java:85) at com.eatthepath.pushy.apns.ApnsClientHandler$ApnsClientHandlerBuilder.build(ApnsClientHandler.java:139) at com.eatthepath.pushy.apns.ApnsClientHandler$ApnsClientHandlerBuilder.build(ApnsClientHandler.java:92) at io.netty.handler.codec.http2.AbstractHttp2ConnectionHandlerBuilder.buildFromCodec(AbstractHttp2ConnectionHandlerBuilder.java:614) at io.netty.handler.codec.http2.AbstractHttp2ConnectionHandlerBuilder.buildFromConnection(AbstractHttp2ConnectionHandlerBuilder.java:589) at io.netty.handler.codec.http2.AbstractHttp2ConnectionHandlerBuilder.build(AbstractHttp2ConnectionHandlerBuilder.java:553) at com.eatthepath.pushy.apns.ApnsClientHandler$ApnsClientHandlerBuilder.build(ApnsClientHandler.java:146) at com.eatthepath.pushy.apns.ApnsChannelFactory$1.initChannel(ApnsChannelFactory.java:129) at com.eatthepath.pushy.apns.ApnsChannelFactory$1.initChannel(ApnsChannelFactory.java:97) at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129) at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112) at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:1114) at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609) at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46) at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463) at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115) at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650) at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514) at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:480) at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:89) at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:83) at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:339) at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:164) at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:125) at com.eatthepath.pushy.apns.ApnsChannelFactory.lambda$create$3(ApnsChannelFactory.java:175) at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153) at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829)

[pgus2024]

Along with the above error, this error is also seen in spikes:

Caused by: io.netty.handler.codec.http2.Http2Exception$StreamException: Stream closed before write could take place at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:173) at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$FlowState.cancel(DefaultHttp2RemoteFlowController.java:481) at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$1.onStreamClosed(DefaultHttp2RemoteFlowController.java:105) at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:357) at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1007) at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:963) at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:515) at io.netty.handler.codec.http2.DefaultHttp2Connection.close(DefaultHttp2Connection.java:153) at io.netty.handler.codec.http2.Http2ConnectionHandler$BaseDecoder.channelInactive(Http2ConnectionHandler.java:217) at io.netty.handler.codec.http2.Http2ConnectionHandler.channelInactive(Http2ConnectionHandler.java:432) at com.eatthepath.pushy.apns.ApnsClientHandler.channelInactive(ApnsClientHandler.java:525) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:305) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:411) at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:376) at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1174) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:305) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813) at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)