Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZClient cannot recover after a network failure or connection timeouts #2653

Closed
mikail-khan opened this issue Jan 25, 2024 · 17 comments
Closed
Labels
💎 Bounty bug Something isn't working 💰 Rewarded

Comments

@mikail-khan
Copy link

Describe the bug
When a network recovers after a temporary failure, a running ZClient program with a retry schedule policy specified fails to determine that the network connection has recovered and continues retrying with the following error message:

[2024-01-24T22:53:48,110]  [i.n.h.s.u.InsecureTrustManagerFactory] [DEBUG:] - Accepting a server certificate: CN=xxx.xxx.com
[2024-01-24T22:53:48,113]  [i.n.h.s.SslHandler] [DEBUG:] - [id: 0x6ce64463, L:/xxx.xxx.xxx.xxx - R:xxx.xxx.com] HANDSHAKEN: protocol:TLSv1.3 cipher suite:TLS_AES_128_GCM_SHA256
[2024-01-24T22:54:04,108]  [b.s.T.Live] [ERROR:] - : Could not get system status - null: xxx.xxx.com/xxx.xxx:443.
[2024-01-24T22:54:04,109]  [<.M.program] [ERROR:] - : null: xxx.xxx.com/xxx.xxx:443.
[2024-01-24T22:54:04,108]  [b.h.K.Live] [DEBUG:] - Fiber zio-fiber-250 did not handle an error
 io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: xxx.xxx.com/xxx.xxx:443
Caused by: java.net.NoRouteToHostException
	at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.handler.ssl.SslHandler.connect(SslHandler.java:737) ~[netty-handler-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:657) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
	at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
[2024-01-24T22:54:04,128]  [i.n.h.s.u.InsecureTrustManagerFactory] [DEBUG:] - Accepting a server certificate: CN=xxx.xxx.com
[2024-01-24T22:54:04,130]  [i.n.h.s.SslHandler] [DEBUG:] - [id: 0x16d75050, L:/xxx.xxx.xxx.xxx - R:xxx.xxx.com/xxx.xxx:443] HANDSHAKEN: protocol:TLSv1.3 cipher suite:TLS_AES_128_GCM_SHA256

To Reproduce
Steps to reproduce the behaviour:

  1. Create a simple ZClient program which makes repeat HTTP requests and has a schedule policy defined
  2. Run the program and observe that the HTTP requests are being made successfully
  3. Kill the network and observe in the logs the following exception: java.net.NoRouteToHostException
  4. Observe that the exponential back-off schedule policy kicks in and retries the request based on the schedule policy.
  5. Restore the network
  6. Observe that ZClient fails to determine the network is up and the request is continuously retried but fails.

Expected behaviour
On network recovery, the request is retried and succeeds.

@mikail-khan mikail-khan added the bug Something isn't working label Jan 25, 2024
@kyri-petrou
Copy link
Collaborator

@mikail-khan I believe this might have been resolved by #2610. Can you try with the latest snapshot (3.0.0-RC4+58-a70a964c-SNAPSHOT) and see if it's still an issue?

@mikail-khan
Copy link
Author

mikail-khan commented Jan 25, 2024

3.0.0-RC4+58-a70a964c-SNAPSHOT

I created a small repo program below to demonstrate the problem and so you can debug this as well.

On a java.net.NoRouteToHostException the program fails to recover after the network recovers. With the snapshot version, I recovered the network at 2024-01-25T10:52.00, but the fiber does not detect it and continues retrying.

import zio._

import zio.http._
import zio.logging.backend.SLF4J
import java.util.concurrent.TimeUnit

object SimpleClient extends ZIOAppDefault {

  override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] =
    Runtime.removeDefaultLoggers >>> SLF4J.slf4j

  val url = URL.decode("http://www.google.com").toOption.get

  val program = for {
    client <- ZIO.service[Client]
    res    <- client.url(url).get("/")
    data   <- res.body.asString
    _      <- ZIO.logInfo(s"Got data: ${data.take(30)}")
  } yield ()

  override val run = program
                      .tapError(_ => ZIO.logError(s"boom!"))
                      .repeat(Schedule.spaced(Duration(3, TimeUnit.SECONDS)))
                      .retry(Schedule.exponential(Duration(1, TimeUnit.SECONDS)))
                      .provide(Client.default, Scope.default)

}
3.0.0-RC4+58-a70a964c-SNAPSHOT log output

[2024-01-25T10:50:15,420] [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.chunkSize: 32
[2024-01-25T10:50:15,421] [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.blocking: false
[2024-01-25T10:50:15,422] [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.batchFastThreadLocalOnly: true
[2024-01-25T10:50:15,666] [<.SimpleClient] [INFO :] - Got data: <!doctype html><html itemscope="" itemtype="http:/
[2024-01-25T10:50:18,782] [<.SimpleClient] [INFO :] - Got data: <!doctype html><html itemscope="" itemtype="http:/
[2024-01-25T10:50:22,003] [<.SimpleClient] [INFO :] - Got data: <!doctype html><html itemscope="" itemtype="http:/
[2024-01-25T10:51:05,397] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-01-25T10:51:05,402] [<.SimpleClient] [ERROR:] - boom!
[2024-01-25T10:51:05,447] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-01-25T10:51:05,462] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-01-25T10:51:05,474] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-01-25T10:51:05,483] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-01-25T10:51:05,489] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-01-25T10:51:05,507] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-01-25T10:51:05,619] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-01-25T10:51:08,783] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-01-25T10:51:08,786] [<.SimpleClient] [ERROR:] - boom!
[2024-01-25T10:51:12,002] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-01-25T10:51:12,004] [<.SimpleClient] [ERROR:] - boom!
[2024-01-25T10:51:16,025] [<.SimpleClient] [ERROR:] - boom!
[2024-01-25T10:51:16,033] [<.SimpleClient] [DEBUG:] - Fiber zio-fiber-105 did not handle an error
io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: www.google.com/216.239.38.120:80
Caused by: java.net.NoRouteToHostException
at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
[2024-01-25T10:51:24,046] [<.SimpleClient] [ERROR:] - boom!
[2024-01-25T10:51:24,045] [<.SimpleClient] [DEBUG:] - Fiber zio-fiber-107 did not handle an error
io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: www.google.com/216.239.38.120:80
Caused by: java.net.NoRouteToHostException
at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
[2024-01-25T10:51:40,064] [<.SimpleClient] [ERROR:] - boom!
[2024-01-25T10:51:40,064] [<.SimpleClient] [DEBUG:] - Fiber zio-fiber-109 did not handle an error
io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: www.google.com/216.239.38.120:80
Caused by: java.net.NoRouteToHostException
at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
[2024-01-25T10:52:12,083] [<.SimpleClient] [DEBUG:] - Fiber zio-fiber-111 did not handle an error
io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: www.google.com/216.239.38.120:80
Caused by: java.net.NoRouteToHostException
at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
[2024-01-25T10:52:12,085] [<.SimpleClient] [ERROR:] - boom!
[2024-01-25T10:52:14,093] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-01-25T10:52:30,113] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-01-25T10:53:02,129] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught

@mikail-khan
Copy link
Author

mikail-khan commented Jan 25, 2024

same issue for connection timeouts, program fails to recover after network recovery

logs [2024-01-25T21:28:31,817] [b.h.K.Live] [DEBUG:] - Fiber zio-fiber-69763 did not handle an error io.netty.channel.ConnectTimeoutException: connection timed out: xxx.xxx.xxx:443 at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$2.run(AbstractEpollChannel.java:613) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:416) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]

Cannot build a resilient service if retries don't succeed when they should.

@mikail-khan mikail-khan changed the title ZClient cannot recover after a network failure ZClient cannot recover after a network failure or connection timeouts Jan 25, 2024
@kyri-petrou
Copy link
Collaborator

kyri-petrou commented Jan 25, 2024

@mikail-khan I'm away for the weekend so I don't have access to my laptop so I can't test it, but could you try running it via ZIO.scoped(program), with the retries / error handling applied outside of it? I think the scope might need to be closed prior to retrying otherwise the client won't release the invalid connection.

Note, that means you should not need Scope.default as a layer to your program

@jdegoes
Copy link
Member

jdegoes commented Feb 8, 2024

/bounty $150

Copy link

algora-pbc bot commented Feb 8, 2024

💎 $150 bounty • ZIO

Steps to solve:

  1. Start working: Comment /attempt #2653 with your implementation plan
  2. Submit work: Create a pull request including /claim #2653 in the PR body to claim the bounty
  3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to zio/zio-http!

Add a bountyShare on socials

Attempt Started (GMT+0) Solution
🔴 @TheVixhal Feb 10, 2024, 10:24:00 AM WIP
🔴 @asr2003 Jun 22, 2024, 6:45:18 PM WIP
🟢 @kyri-petrou #3008

@TheVixhal
Copy link

TheVixhal commented Feb 10, 2024

/attempt #2653

@kyri-petrou
Copy link
Collaborator

@mikail-khan I can't seem to be able to reproduce the issue you're facing. With the current main, and the code below, I get the following console output after I kill the network and enable it again:

Code:

object SimpleClient extends ZIOAppDefault {

  val url = URL.decode("http://www.google.com").toOption.get

  val program = for {
    client <- ZIO.service[Client]
    res    <- client.url(url).get("/")
    data   <- res.body.asString
    _      <- ZIO.debug(s"Got data: ${data.take(30)}")
  } yield ()

  override val run =
    ZIO
      .scoped(program)
      .tapError(_ => ZIO.debug(s"boom!"))
      .repeat(Schedule.spaced(Duration(3, TimeUnit.SECONDS)))
      .retry(Schedule.fixed(Duration(1, TimeUnit.SECONDS)))
      .provide(Client.default)
}

Output:

Got data: <!doctype html><html itemscope
Got data: <!doctype html><html itemscope
Got data: <!doctype html><html itemscope
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
Got data: <!doctype html><html itemscope
Got data: <!doctype html><html itemscope
Got data: <!doctype html><html itemscope
Got data: <!doctype html><html itemscope

@mikail-khan
Copy link
Author

mikail-khan commented Feb 13, 2024

@mikail-khan I can't seem to be able to reproduce the issue you're facing.

See below, I've reproduced it again using the program I posted earlier. At 21:14 the network was recovered but the program does not detect it.

"dev.zio" %% "zio-http" % "3.0.0-RC4+58-a70a964c-SNAPSHOT",

[2024-02-13T21:12:14,559]  [i.n.u.i.l.InternalLoggerFactory] [DEBUG:] - Using SLF4J as the default logging framework
[2024-02-13T21:12:14,602]  [i.n.u.i.PlatformDependent0] [DEBUG:] - -Dio.netty.noUnsafe: false
[2024-02-13T21:12:14,604]  [i.n.u.i.PlatformDependent0] [DEBUG:] - Java version: 17
[2024-02-13T21:12:14,606]  [i.n.u.i.PlatformDependent0] [DEBUG:] - sun.misc.Unsafe.theUnsafe: available
[2024-02-13T21:12:14,607]  [i.n.u.i.PlatformDependent0] [DEBUG:] - sun.misc.Unsafe.copyMemory: available
[2024-02-13T21:12:14,608]  [i.n.u.i.PlatformDependent0] [DEBUG:] - sun.misc.Unsafe.storeFence: available
[2024-02-13T21:12:14,609]  [i.n.u.i.PlatformDependent0] [DEBUG:] - java.nio.Buffer.address: available
[2024-02-13T21:12:14,610]  [i.n.u.i.PlatformDependent0] [DEBUG:] - direct buffer constructor: unavailable: Reflective setAccessible(true) disabled
[2024-02-13T21:12:14,611]  [i.n.u.i.PlatformDependent0] [DEBUG:] - java.nio.Bits.unaligned: available, true
[2024-02-13T21:12:14,613]  [i.n.u.i.PlatformDependent0] [DEBUG:] - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable: class io.netty.util.internal.PlatformDependent0$7 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @768b0f44
[2024-02-13T21:12:14,615]  [i.n.u.i.PlatformDependent0] [DEBUG:] - java.nio.DirectByteBuffer.<init>(long, {int,long}): unavailable
[2024-02-13T21:12:14,615]  [i.n.u.i.PlatformDependent] [DEBUG:] - sun.misc.Unsafe: available
[2024-02-13T21:12:14,616]  [i.n.u.i.PlatformDependent] [DEBUG:] - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
[2024-02-13T21:12:14,616]  [i.n.u.i.PlatformDependent] [DEBUG:] - -Dio.netty.bitMode: 64 (sun.arch.data.model)
[2024-02-13T21:12:14,621]  [i.n.u.i.PlatformDependent] [DEBUG:] - -Dio.netty.maxDirectMemory: -1 bytes
[2024-02-13T21:12:14,623]  [i.n.u.i.PlatformDependent] [DEBUG:] - -Dio.netty.uninitializedArrayAllocationThreshold: -1
[2024-02-13T21:12:14,630]  [i.n.u.i.CleanerJava9] [DEBUG:] - java.nio.ByteBuffer.cleaner(): available
[2024-02-13T21:12:14,632]  [i.n.u.i.PlatformDependent] [DEBUG:] - -Dio.netty.noPreferDirect: false
[2024-02-13T21:12:14,647]  [i.n.u.i.NativeLibraryLoader] [DEBUG:] - -Dio.netty.native.workdir: /tmp (io.netty.tmpdir)
[2024-02-13T21:12:14,648]  [i.n.u.i.NativeLibraryLoader] [DEBUG:] - -Dio.netty.native.deleteLibAfterLoading: true
[2024-02-13T21:12:14,649]  [i.n.u.i.NativeLibraryLoader] [DEBUG:] - -Dio.netty.native.tryPatchShadedId: true
[2024-02-13T21:12:14,650]  [i.n.u.i.NativeLibraryLoader] [DEBUG:] - -Dio.netty.native.detectNativeLibraryDuplicates: true
[2024-02-13T21:12:14,665]  [i.n.u.i.NativeLibraryLoader] [DEBUG:] - Successfully loaded the library /tmp/libnetty_transport_native_epoll_x86_643484322894985639787.so
[2024-02-13T21:12:14,673]  [i.n.u.NetUtil] [DEBUG:] - -Djava.net.preferIPv4Stack: false
[2024-02-13T21:12:14,675]  [i.n.u.NetUtil] [DEBUG:] - -Djava.net.preferIPv6Addresses: false
[2024-02-13T21:12:14,682]  [i.n.u.NetUtilInitializations] [DEBUG:] - Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo)
[2024-02-13T21:12:14,684]  [i.n.u.NetUtil] [DEBUG:] - /proc/sys/net/core/somaxconn: 4096
[2024-02-13T21:12:14,703]  [i.n.c.MultithreadEventLoopGroup] [DEBUG:] - -Dio.netty.eventLoopThreads: 8
[2024-02-13T21:12:14,726]  [i.n.u.c.GlobalEventExecutor] [DEBUG:] - -Dio.netty.globalEventExecutor.quietPeriodSeconds: 1
[2024-02-13T21:12:14,743]  [i.n.u.i.InternalThreadLocalMap] [DEBUG:] - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
[2024-02-13T21:12:14,745]  [i.n.u.i.InternalThreadLocalMap] [DEBUG:] - -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
[2024-02-13T21:12:14,773]  [i.n.u.i.PlatformDependent] [DEBUG:] - org.jctools-core.MpscChunkedArrayQueue: available
[2024-02-13T21:12:15,019]  [i.n.c.DefaultChannelId] [DEBUG:] - -Dio.netty.processId: 134982 (auto-detected)
[2024-02-13T21:12:15,026]  [i.n.c.DefaultChannelId] [DEBUG:] - -Dio.netty.machineId: 5c:e0:c5:ff:fe:51:f5:ef (auto-detected)
[2024-02-13T21:12:15,048]  [i.n.u.ResourceLeakDetector] [DEBUG:] - -Dio.netty.leakDetection.level: simple
[2024-02-13T21:12:15,050]  [i.n.u.ResourceLeakDetector] [DEBUG:] - -Dio.netty.leakDetection.targetRecords: 4
[2024-02-13T21:12:15,098]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.numHeapArenas: 8
[2024-02-13T21:12:15,098]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.numDirectArenas: 8
[2024-02-13T21:12:15,099]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.pageSize: 8192
[2024-02-13T21:12:15,099]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.maxOrder: 9
[2024-02-13T21:12:15,100]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.chunkSize: 4194304
[2024-02-13T21:12:15,100]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.smallCacheSize: 256
[2024-02-13T21:12:15,101]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.normalCacheSize: 64
[2024-02-13T21:12:15,102]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
[2024-02-13T21:12:15,103]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.cacheTrimInterval: 8192
[2024-02-13T21:12:15,104]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.cacheTrimIntervalMillis: 0
[2024-02-13T21:12:15,107]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.useCacheForAllThreads: false
[2024-02-13T21:12:15,108]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
[2024-02-13T21:12:15,133]  [i.n.b.ByteBufUtil] [DEBUG:] - -Dio.netty.allocator.type: pooled
[2024-02-13T21:12:15,134]  [i.n.b.ByteBufUtil] [DEBUG:] - -Dio.netty.threadLocalDirectBufferSize: 0
[2024-02-13T21:12:15,134]  [i.n.b.ByteBufUtil] [DEBUG:] - -Dio.netty.maxThreadLocalCharBufferSize: 16384
[2024-02-13T21:12:15,165]  [i.n.b.ChannelInitializerExtensions] [DEBUG:] - -Dio.netty.bootstrap.extensions: null
[2024-02-13T21:12:15,224]  [i.n.b.AbstractByteBuf] [DEBUG:] - -Dio.netty.buffer.checkAccessible: true
[2024-02-13T21:12:15,226]  [i.n.b.AbstractByteBuf] [DEBUG:] - -Dio.netty.buffer.checkBounds: true
[2024-02-13T21:12:15,228]  [i.n.u.ResourceLeakDetectorFactory] [DEBUG:] - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@6c06e93
[2024-02-13T21:12:15,350]  [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.maxCapacityPerThread: 4096
[2024-02-13T21:12:15,351]  [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.ratio: 8
[2024-02-13T21:12:15,351]  [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.chunkSize: 32
[2024-02-13T21:12:15,351]  [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.blocking: false
[2024-02-13T21:12:15,352]  [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.batchFastThreadLocalOnly: true
[2024-02-13T21:12:15,602]  [<.SimpleClient] [INFO :] - Got data: <!doctype html><html itemscope="" itemtype="http:/
[2024-02-13T21:12:18,700]  [<.SimpleClient] [INFO :] - Got data: <!doctype html><html itemscope="" itemtype="http:/
[2024-02-13T21:12:22,165]  [<.SimpleClient] [INFO :] - Got data: <!doctype html><html itemscope="" itemtype="http:/
[2024-02-13T21:13:05,348]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:13:05,361]  [<.SimpleClient] [ERROR:] - boom!
[2024-02-13T21:13:05,374]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:13:05,389]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:13:05,394]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:13:05,406]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:13:05,409]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:13:05,425]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:13:05,570]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:13:08,698]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:13:08,700]  [<.SimpleClient] [ERROR:] - boom!
[2024-02-13T21:13:12,166]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:13:12,171]  [<.SimpleClient] [ERROR:] - boom!
[2024-02-13T21:13:16,184]  [<.SimpleClient] [ERROR:] - boom!
[2024-02-13T21:13:16,186]  [<.SimpleClient] [DEBUG:] - Fiber zio-fiber-108 did not handle an error
 io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: www.google.com/216.239.38.120:80
Caused by: java.net.NoRouteToHostException
        at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
[2024-02-13T21:13:24,191]  [<.SimpleClient] [ERROR:] - boom!
[2024-02-13T21:13:24,191]  [<.SimpleClient] [DEBUG:] - Fiber zio-fiber-110 did not handle an error
 io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: www.google.com/216.239.38.120:80
Caused by: java.net.NoRouteToHostException
        at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
[2024-02-13T21:13:40,196]  [<.SimpleClient] [ERROR:] - boom!
[2024-02-13T21:13:40,196]  [<.SimpleClient] [DEBUG:] - Fiber zio-fiber-112 did not handle an error
 io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: www.google.com/216.239.38.120:80
Caused by: java.net.NoRouteToHostException
        at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
[2024-02-13T21:14:12,215]  [<.SimpleClient] [ERROR:] - boom!
[2024-02-13T21:14:12,215]  [<.SimpleClient] [DEBUG:] - Fiber zio-fiber-114 did not handle an error
 io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: www.google.com/216.239.38.120:80
Caused by: java.net.NoRouteToHostException
        at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
[2024-02-13T21:15:02,262]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:15:16,296]  [<.SimpleClient] [ERROR:] - boom!
[2024-02-13T21:15:16,297]  [<.SimpleClient] [DEBUG:] - Fiber zio-fiber-117 did not handle an error
 io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: www.google.com/216.239.38.120:80
Caused by: java.net.NoRouteToHostException
        at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
[2024-02-13T21:16:06,345]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:17:24,314]  [<.SimpleClient] [ERROR:] - boom!
[2024-02-13T21:17:24,314]  [<.SimpleClient] [DEBUG:] - Fiber zio-fiber-120 did not handle an error
 io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: www.google.com/216.239.38.120:80
Caused by: java.net.NoRouteToHostException
        at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
[2024-02-13T21:12:14,559] [i.n.u.i.l.InternalLoggerFactory] [DEBUG:] - Using SLF4J as the default logging framework [2024-02-13T21:12:14,602] [i.n.u.i.PlatformDependent0] [DEBUG:] - -Dio.netty.noUnsafe: false [2024-02-13T21:12:14,604] [i.n.u.i.PlatformDependent0] [DEBUG:] - Java version: 17 [2024-02-13T21:12:14,606] [i.n.u.i.PlatformDependent0] [DEBUG:] - sun.misc.Unsafe.theUnsafe: available [2024-02-13T21:12:14,607] [i.n.u.i.PlatformDependent0] [DEBUG:] - sun.misc.Unsafe.copyMemory: available [2024-02-13T21:12:14,608] [i.n.u.i.PlatformDependent0] [DEBUG:] - sun.misc.Unsafe.storeFence: available [2024-02-13T21:12:14,609] [i.n.u.i.PlatformDependent0] [DEBUG:] - java.nio.Buffer.address: available [2024-02-13T21:12:14,610] [i.n.u.i.PlatformDependent0] [DEBUG:] - direct buffer constructor: unavailable: Reflective setAccessible(true) disabled [2024-02-13T21:12:14,611] [i.n.u.i.PlatformDependent0] [DEBUG:] - java.nio.Bits.unaligned: available, true [2024-02-13T21:12:14,613] [i.n.u.i.PlatformDependent0] [DEBUG:] - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable: class io.netty.util.internal.PlatformDependent0$7 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @768b0f44 [2024-02-13T21:12:14,615] [i.n.u.i.PlatformDependent0] [DEBUG:] - java.nio.DirectByteBuffer.(long, {int,long}): unavailable [2024-02-13T21:12:14,615] [i.n.u.i.PlatformDependent] [DEBUG:] - sun.misc.Unsafe: available [2024-02-13T21:12:14,616] [i.n.u.i.PlatformDependent] [DEBUG:] - -Dio.netty.tmpdir: /tmp (java.io.tmpdir) [2024-02-13T21:12:14,616] [i.n.u.i.PlatformDependent] [DEBUG:] - -Dio.netty.bitMode: 64 (sun.arch.data.model) [2024-02-13T21:12:14,621] [i.n.u.i.PlatformDependent] [DEBUG:] - -Dio.netty.maxDirectMemory: -1 bytes [2024-02-13T21:12:14,623] [i.n.u.i.PlatformDependent] [DEBUG:] - -Dio.netty.uninitializedArrayAllocationThreshold: -1 [2024-02-13T21:12:14,630] [i.n.u.i.CleanerJava9] [DEBUG:] - java.nio.ByteBuffer.cleaner(): available [2024-02-13T21:12:14,632] [i.n.u.i.PlatformDependent] [DEBUG:] - -Dio.netty.noPreferDirect: false [2024-02-13T21:12:14,647] [i.n.u.i.NativeLibraryLoader] [DEBUG:] - -Dio.netty.native.workdir: /tmp (io.netty.tmpdir) [2024-02-13T21:12:14,648] [i.n.u.i.NativeLibraryLoader] [DEBUG:] - -Dio.netty.native.deleteLibAfterLoading: true [2024-02-13T21:12:14,649] [i.n.u.i.NativeLibraryLoader] [DEBUG:] - -Dio.netty.native.tryPatchShadedId: true [2024-02-13T21:12:14,650] [i.n.u.i.NativeLibraryLoader] [DEBUG:] - -Dio.netty.native.detectNativeLibraryDuplicates: true [2024-02-13T21:12:14,665] [i.n.u.i.NativeLibraryLoader] [DEBUG:] - Successfully loaded the library /tmp/libnetty_transport_native_epoll_x86_643484322894985639787.so [2024-02-13T21:12:14,673] [i.n.u.NetUtil] [DEBUG:] - -Djava.net.preferIPv4Stack: false [2024-02-13T21:12:14,675] [i.n.u.NetUtil] [DEBUG:] - -Djava.net.preferIPv6Addresses: false [2024-02-13T21:12:14,682] [i.n.u.NetUtilInitializations] [DEBUG:] - Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo) [2024-02-13T21:12:14,684] [i.n.u.NetUtil] [DEBUG:] - /proc/sys/net/core/somaxconn: 4096 [2024-02-13T21:12:14,703] [i.n.c.MultithreadEventLoopGroup] [DEBUG:] - -Dio.netty.eventLoopThreads: 8 [2024-02-13T21:12:14,726] [i.n.u.c.GlobalEventExecutor] [DEBUG:] - -Dio.netty.globalEventExecutor.quietPeriodSeconds: 1 [2024-02-13T21:12:14,743] [i.n.u.i.InternalThreadLocalMap] [DEBUG:] - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024 [2024-02-13T21:12:14,745] [i.n.u.i.InternalThreadLocalMap] [DEBUG:] - -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096 [2024-02-13T21:12:14,773] [i.n.u.i.PlatformDependent] [DEBUG:] - org.jctools-core.MpscChunkedArrayQueue: available [2024-02-13T21:12:15,019] [i.n.c.DefaultChannelId] [DEBUG:] - -Dio.netty.processId: 134982 (auto-detected) [2024-02-13T21:12:15,026] [i.n.c.DefaultChannelId] [DEBUG:] - -Dio.netty.machineId: 5c:e0:c5:ff:fe:51:f5:ef (auto-detected) [2024-02-13T21:12:15,048] [i.n.u.ResourceLeakDetector] [DEBUG:] - -Dio.netty.leakDetection.level: simple [2024-02-13T21:12:15,050] [i.n.u.ResourceLeakDetector] [DEBUG:] - -Dio.netty.leakDetection.targetRecords: 4 [2024-02-13T21:12:15,098] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.numHeapArenas: 8 [2024-02-13T21:12:15,098] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.numDirectArenas: 8 [2024-02-13T21:12:15,099] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.pageSize: 8192 [2024-02-13T21:12:15,099] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.maxOrder: 9 [2024-02-13T21:12:15,100] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.chunkSize: 4194304 [2024-02-13T21:12:15,100] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.smallCacheSize: 256 [2024-02-13T21:12:15,101] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.normalCacheSize: 64 [2024-02-13T21:12:15,102] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.maxCachedBufferCapacity: 32768 [2024-02-13T21:12:15,103] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.cacheTrimInterval: 8192 [2024-02-13T21:12:15,104] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.cacheTrimIntervalMillis: 0 [2024-02-13T21:12:15,107] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.useCacheForAllThreads: false [2024-02-13T21:12:15,108] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023 [2024-02-13T21:12:15,133] [i.n.b.ByteBufUtil] [DEBUG:] - -Dio.netty.allocator.type: pooled [2024-02-13T21:12:15,134] [i.n.b.ByteBufUtil] [DEBUG:] - -Dio.netty.threadLocalDirectBufferSize: 0 [2024-02-13T21:12:15,134] [i.n.b.ByteBufUtil] [DEBUG:] - -Dio.netty.maxThreadLocalCharBufferSize: 16384 [2024-02-13T21:12:15,165] [i.n.b.ChannelInitializerExtensions] [DEBUG:] - -Dio.netty.bootstrap.extensions: null [2024-02-13T21:12:15,224] [i.n.b.AbstractByteBuf] [DEBUG:] - -Dio.netty.buffer.checkAccessible: true [2024-02-13T21:12:15,226] [i.n.b.AbstractByteBuf] [DEBUG:] - -Dio.netty.buffer.checkBounds: true [2024-02-13T21:12:15,228] [i.n.u.ResourceLeakDetectorFactory] [DEBUG:] - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@6c06e93 [2024-02-13T21:12:15,350] [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.maxCapacityPerThread: 4096 [2024-02-13T21:12:15,351] [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.ratio: 8 [2024-02-13T21:12:15,351] [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.chunkSize: 32 [2024-02-13T21:12:15,351] [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.blocking: false [2024-02-13T21:12:15,352] [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.batchFastThreadLocalOnly: true [2024-02-13T21:12:15,602] [<.SimpleClient] [INFO :] - Got data:

@kyri-petrou
Copy link
Collaborator

kyri-petrou commented Feb 13, 2024

@mikail-khan can you try with the latest snapshot (3.0.0-RC4+71-b1da91b6-SNAPSHOT), and the program I provided?

The reason I'm asking to use the program I provided is because it uses ZIO.scoped which is how it's meant to be run (as opposed to providing the scope as a layer)

@mikail-khan
Copy link
Author

mikail-khan commented Feb 14, 2024

@mikail-khan can you try with the latest snapshot (3.0.0-RC4+71-b1da91b6-SNAPSHOT), and the program I provided?

The reason I'm asking to use the program I provided is because it uses ZIO.scoped which is how it's meant to be run (as opposed to providing the scope as a layer)

where does it say ZIO.scoped is "meant" to be run how you say? The example I gave is from the ZIO-HTTP docs.... @jdegoes can you confirm @kyri-petrou 's assertion? IMO, @kyri-petrou 's program doesn't seem idiomatic ZIO.

@kyri-petrou
Copy link
Collaborator

@mikail-khan I'm not sure whether it's idiomatic or not, but it's the correct way to run it. The Scope returned by the get method is responsible for returning the connection back to the connection pool, potentially discarding it if it's dead. Regarding the docs, they should probably be corrected.

Having said that, do we need to have a requirement on Scope? For that I'm not sure. AFAICT its main purpose is so that the caller handles streaming responses properly prior to returning the connection back to the pool, but there might be more UX-friendly (or idiomatic) ways of handling it. Or at the very least the purpose of the scope should be better documented

@asr2003
Copy link
Contributor

asr2003 commented Jun 22, 2024

I will address this by adding the documentation to explain the purpose and correct usage of scopes and why ZIO.scoped is crucial for proper resource management, particularly in network connection scenarios.
/attempt #2653

Copy link

algora-pbc bot commented Jun 29, 2024

@asr2003: Reminder that in 7 days the bounty will become up for grabs, so please submit a pull request before then 🙏

Copy link

algora-pbc bot commented Jul 6, 2024

The bounty is up for grabs! Everyone is welcome to /attempt #2653 🙌

@987Nabil
Copy link
Contributor

Fixed by #3008

Copy link

algora-pbc bot commented Sep 2, 2024

🎉🎈 @kyri-petrou has been awarded $150! 🎈🎊

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💎 Bounty bug Something isn't working 💰 Rewarded
Projects
None yet
Development

No branches or pull requests

6 participants